Databricks RunJob Returning 403: Invalid access token
search cancel

Databricks RunJob Returning 403: Invalid access token

book

Article ID: 266570

calendar_today

Updated On:

Products

Workload Automation Agent

Issue/Introduction

Autusys job is showing failed in production but in databricks we can see its success. The spool for the databricks job has the following:

2023-05-08T19:15:29.355-07:00: Log level: 1
2023-05-08T19:15:29.946-07:00: Job failed: Request failed: 403 - Forbiddencom.ca.http.HttpClientExceptionHC4: Request failed: 403 - Forbidden
@at com.ca.http.HttpClientPojoHC45.processResponse(HttpClientPojoHC45.java:170)
@at com.ca.http.HttpClientPojoHC45.post(HttpClientPojoHC45.java:305)
@at com.broadcom.databricks.DatabricksPojo.runJobPayload(DatabricksPojo.java:882)
@at com.broadcom.databricks.DatabricksPojo.runJob(DatabricksPojo.java:836)
@at com.broadcom.databricks.DatabricksPojo.runJob(DatabricksPojo.java:721)
@at com.broadcom.databricks.DatabricksPojo.runJob(DatabricksPojo.java:681)
@at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
@at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
@at java.lang.reflect.Method.invoke(Method.java:498)
@at cybermation.plugins.dataservices.handler.CybPOJOHandler.a(CybPOJOHandler.java:176)
@at cybermation.plugins.dataservices.handler.CybPOJOHandler.run(CybPOJOHandler.java:268)@at java.lang.Thread.run(Thread.java:821)
 
2023-05-08T19:15:29.946-07:00: Response: {""error_code"":""403"",""message"":""Invalid access token.""}
Execution of runJob has failed
java.lang.reflect.InvocationTargetException: null
@at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
@at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
@at java.lang.reflect.Method.invoke(Method.java:498)
@at cybermation.plugins.dataservices.handler.CybPOJOHandler.a(CybPOJOHandler.java:176)
@at cybermation.plugins.dataservices.handler.CybPOJOHandler.run(CybPOJOHandler.java:268)
@at java.lang.Thread.run(Thread.java:821)Caused by: com.broadcom.databricks.JobFailedException: Request failed: 403 - Forbidden - {""error_code"":""403"",""message"":""Invalid access token.""}
@at com.broadcom.databricks.DatabricksPojo.runJobPayload(DatabricksPojo.java:1003)
@at com.broadcom.databricks.DatabricksPojo.runJob(DatabricksPojo.java:836)
@at com.broadcom.databricks.DatabricksPojo.runJob(DatabricksPojo.java:721)
@at com.broadcom.databricks.DatabricksPojo.runJob(DatabricksPojo.java:681)
@at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
@at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
@at java.lang.reflect.Method.invoke(Method.java:498)
@at cybermation.plugins.dataservices.handler.CybPOJOHandler.a(CybPOJOHandler.java:176)
@at cybermation.plugins.dataservices.handler.CybPOJOHandler.run(CybPOJOHandler.java:268)
@at java.lang.Thread.run(Thread.java:821)Caused by: com.ca.http.HttpClientExceptionHC4: Request failed: 403 - Forbidden
@at com.ca.http.HttpClientPojoHC45.processResponse(HttpClientPojoHC45.java:170)
@at com.ca.http.HttpClientPojoHC45.post(HttpClientPojoHC45.java:305)
@at com.broadcom.databricks.DatabricksPojo.runJobPayload(DatabricksPojo.java:882)
@at com.broadcom.databricks.DatabricksPojo.runJob(DatabricksPojo.java:836)
@at com.broadcom.databricks.DatabricksPojo.runJob(DatabricksPojo.java:721)
@at com.broadcom.databricks.DatabricksPojo.runJob(DatabricksPojo.java:681)
@at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
@at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
@at java.lang.reflect.Method.invoke(Method.java:498)
@at cybermation.plugins.dataservices.handler.CybPOJOHandler.a(CybPOJOHandler.java:176)
@at cybermation.plugins.dataservices.handler.CybPOJOHandler.run(CybPOJOHandler.java:268)
@at java.lang.Thread.run(Thread.java:821)"
 

Environment

Release : 12.3

Cause

Root cause not confirmed. Root cause expected to be related to the token and how Azure is authorizing the token. 

Resolution

Recommendations are included below. If they do not help, please open an issue with Technical Support. The Azure Databricks administrators will need to be involved/available. 

 

Recommendations:

  • Check to see if the token provided has expired? 
  • Check to see if they are getting the same message when manually submitting (via Rest) the call through something like postman.
  • Use the latest version of the plugin extension. See Changes for details on updates to the Amazon S3 Plugin Extension. 
    • The version of the plugin extension being used is often reflected in the jar file found in jars/ext. Ex: jars/ext/databricks12-1.0-11.jar is v1.0-11.jar
    • The latest version of the Plugin Extension can be found here: Databricks Plugin Extension
  • Define jobs using Methods Version 2 and their appropriate parameters as found here: Methods Version 2
  • Enable debug: Enabling debug using Workload Automation Agents v12+ is done through the job. Not the agentparm.txt.

 

Powered by Wolken Software