This techdoc provides an alternative method that you can install Service Desk on SQL named instance, if the original installation hits issues with SQL named instance and non-standard port.
Release : 17.3
You can follow these steps
--install Service Desk using default SQL instance and standard port and verify it works fine after installation
--shutdown SDM and create a named instance and move the mdb to the named instance
--change NX.env file @NX_DB_NODE as
@NX_DB_NODE=SQL-server-name\\instance-name,non-standard-port-number
save and then run pdm_configure
--start SDM if it does not start yet