After applying ZOS maintenance, Zowe fails to start with 401 Unauthorized

Products

• Zowe

Issue/Introduction

After applying z/OS maintenance, Zowe fails to start and reports 401 Unauthorized errors when attempting to communicate with z/OSMF.

The Following errors are observed in Zowe v3:

ZWEAG713E Configuration error when trying to establish JWT producer. Events: Wait for z/OSMF instance ibmzosmf to come online before
 Started waiting for z/OSMF instance ibmzosmf to be registered and known by the discovery service

ZWEAG108E z/OSMF instance 'ibmzosmf' not found or incorrectly configured. ZAAS is shutting down.

 ZWEAG181W z/OSMF service 'ibmzosmf' is either not registered or not online yet.

Request to z/OSMF requires authentication 401 Unauthorized on GET request for  ...

ZWEAG104E Authentication service is not available at URL 'protocol://host.com:port/zosmf/info'

 

 

The Following error is observed in Zowe v2:

Could not contact z/OS MF on 'https://<host:port>/zosmf/info' - 401

 

 

A manual test using curl confirms the issue:

 

Curl -k -iL <protocol>://<host>:<port>/zosmf/info

Result:

 

HTTP/1.1 401 Unauthorized

....

Environment

Zowe Base(Multiple Versions)

Cause

After applying RSU2512, z/OSMF returns 401 Unauthorized responses to /zosmf/info.

According to IBM z/OSMF architect:  APAR PH66874 PTFs UO05428/UO05981 for ZOS 3.2 and UO05331/UO05989 for ZOS 3.1 in RSU2512 changed the authentication requirements for the /info call.

Zowe uses /zosmf/info endpoint during startup to validate z/OSMF availability. Due to the change in z/OSMF behavior, this validation fails even though z/OSMF is running, causing Zowe startup to abort.

Resolution

Zowe 3.4.1 and 2.18.4 are now available with critical fixes for z/OSMF compatibility. 

 

Additional Information

Note the following 2 items are all just workarounds 

1. Modify Zowe validation logic

For Zowe versions prior to 3.4, update the Zowe runtime code to allow 401 return code in validation.

• • <zowe runtime>/bin/libs/zosmf.js - lines 36-37
    
           else if (execReturn.out != '200') {
           else if (execReturn.out != '200' && execReturn.out != '401')
                common.printError(Could not contact z/OS MF on 'https://${zosmfHost}:${zosmfPort}/zosmf/info' - ${execReturn.out});
    
    

    (second file is used only when configManager is turned off)
    <zowe runtime>/bin/libs/zosmf.js/bin/libs/zosmf.sh - lines 43-44

        elif [ ${http_response_code} != 200 ]; then
        else if (execReturn.out != '200' && execReturn.out != '401'); then
          print_error "Could not contact z/OS MF on 'https://${zosmf_host}:${zosmf_port}/zosmf/info' - ${http_response_code}"
    
    

For Zowe versions 3.4 and above, this functionality is officially supported and configurable via zowe.yaml:

zowe:
  launchScript:
    # Startup checks which run as part of the Zowe STC, prior to Zowe component startup.
    startupChecks:
      # Default check behavior may be one of "exit", "warn", or "disabled"
      default: "exit"
      # z/OSMF startup check - defaults to zowe.launchScript.startupChecks.default, or "exit" if this is unset
      zosmf: "warn"

• • Set 'warn' value for the z/OSMF check. All other stay on 'exit'.
  • Note: These changes still do the validation(for z/OSMF availability) and prints in the error in the log, but allows the startup process to continue.

For all ZOWE v3 installations, change the authentication provider to SAF. ZAAS component also uses the info endpoint when zosmf is used as a provider.

components.gateway.apiml.security.auth.provider: saf

Additionally, for ZOWE v3  prior to v3.4, there are two possible files that need to be modified depending on whenever modulith is used. Both can be changed.

<runtime_dir>/components/apiml/zosmf-static-definition.yaml.template
<runtime_dir>/components/discovery/zosmf-static-definition.yaml.template

Update the authentication section as follows:

services:
      authentication:
          scheme: zosmf
          scheme: httpBasicPassTicket
          applid: IZUDFLT #(z/OSMF applid)
After changing the template, restart Zowe and the static definition file in <workspace_dir>/api-mediation/api-defs should update with new values.
 
This change will make z/OSMF calls authenticate using PassTickets instead of a JWT/LTPA token. As such, ensure that PassTickets are configured and enabled for z/OSMF.

2. Configure Zowe to use different z/OSMF without the maintenance.