Use Case: During testing of POJO jobs it was discovered that some reported failure in autosys although they were completed successfully in databricks for example. After adjusting the values, the job is reporting as success in both autosys and databricks. This would need to be addressed on a case by case for each job as they all may have different run times. It is anticipated that many more jobs will be added, and we would like to have a static value; or some infinite value, that could be used as a threshold for all jobs. This would decrease effort and time spent, and would allow for some jobs to run longer than expected without producing failures or requiring JIL changes on the fly. Attempted using different values but some jobs will run 30 seconds and other may run 60+ minutes.
databricks.MaxRetry | The number of retries for the monitor status request in case of failure. |
databricks.SleepInterval | The interval in milliseconds before resubmitting the monitor status request after a failure. |
databricks.FailureInterval | The amount of time in seconds before marking the job as failed due to Databricks REST API failure. |
iics.SleepInterval | The time interval, in seconds, before resubmitting a monitor status request. |
iics.FailureInterval | The time interval, in seconds, before a job fails due to API failure. |
iics.MaxRetry | The maximum number of retries for monitoring status requests in case of failures. |
JIL Examples:
insert_job: DATABRICK_POJO_BRAD job_type: POJO
machine: localhost
owner: autosys
permission:
date_conditions: 0
description: "Testing"
alarm_if_fail: 1
alarm_if_terminated: 1
method_name: runJob
j2ee_parameter: databricks.Endpoint=https\://<host>.azuredatabricks.net
j2ee_parameter: databricks.Token=$$DATA
j2ee_parameter: databricks.JobId=112233445566771
j2ee_parameter: databricks.Parameters="\"notebook_params\":{\"parm1\":\"hello-2025\"}"
j2ee_parameter: databricks.ProxyHost=""
j2ee_parameter: databricks.ProxyPort=""
j2ee_parameter: databricks.ProxyUser=""
j2ee_parameter: databricks.ProxyPassword=""
j2ee_parameter: databricks.MaxRetry=500
j2ee_parameter: databricks.SleepInterval=1000
j2ee_parameter: databricks.FailureInterval=300
j2ee_parameter: databricks.LogLevel=16
class_name: com.broadcom.databricks.DatabricksPojo
insert_job: INFORMATICA_POJO_GLOBAL job_type: POJO
machine: localhost
owner: autosys
permission:
date_conditions: 0
alarm_if_fail: 1
alarm_if_terminated: 1
method_name: startAndMonitorTaskFlow
j2ee_parameter: iics.AuthURL=https\://<host>.informaticacloud.com
j2ee_parameter: iics.Username=some_user_account
j2ee_parameter: iics.Password=$$INFO_PW
j2ee_parameter: iics.ServiceURL=https\://<host>.informaticacloud.com
j2ee_parameter: iics.TaskflowName=tf_generic_source_overwrite_v1
j2ee_parameter: iics.UseParamSet=false
j2ee_parameter: iics.InputFields=""
j2ee_parameter: iics.FailSuspended=false
j2ee_parameter: iics.SleepInterval=5
j2ee_parameter: iics.FailureInterval=5
j2ee_parameter: iics.MaxRetry=10
j2ee_parameter: iics.LogLevel=16
class_name: com.broadcom.pojo.iics.IICS
AutoSys Workload Automation r12.x, r24.x
Workload Automation System Agent r12.x r24.x
The values provided in the Job Definition are converted to java primitive types values internally.
For example, if we provide the value as 2 for 'databricks.MaxRetry', it will be converted to java.lang.Integer value.
While infinity values cannot be provided, consider the maximum allowed values.
Integer.MAX_VALUE = 2147483647
Please note that it is not recommended to use max values.
Taking all of the above into consideration, it is recommend to set the values accordingly for each job so as not to introduce unwanted/unnecessary delays in your batch cycles.
It is important to note that for SleepInterval
there is an unfortunate discrepancy between the different plugins. Some uses seconds and some use milliseconds.
Another important note for consideration; since these are cloud services, there are maintenance windows in which the vendor can make your service unavailable while applying maintenance on the application. Every cloud vendor, service, and SAAS apps have maintenance windows. Consult the vendor of the cloud service regarding contractual maintenance windows.