As per section "Backing up data sets being accessed with record level sharing" on RLS processing in the DFSMSdss Storage Administration V2R3 manual,
DFDSS invoked by BACKUPCC performs the QUIESCE for SMS-managed VSAM RLS files, see link - https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.adru000/rlsbku.htm Backing up data sets being accessed with record level sharing During logical data set dump operations of SMS-managed VSAM data sets, DFSMSdss communicates with VSAM RLS to perform quiesce processing of data sets that are being accessed by another job using Record Level Sharing (RLS).
By default, DFSMSdss does not use timeout protection during RLS quiesce processing.
You can control whether or not DFSMSdss uses timeout protection during RLS quiesce processing and what the timeout value should be using the DSSTIMEOUT parameter of the IGDSMSxx PARMLIB member.
You can also change the timeout value without IPLing the system using the SETSMS DSSTIMEOUT(nnnnn) command.
For more information about the RLS timeout value used during DFSMSdss operations, refer to z/OS DFSMSdfp Storage Administration.
For more information about the SETSMS command, refer to z/OS MVS System Commands.