You are using the MDM capability introduced with 8.6 RU2 and need to request the signing of your certificate as indicated in Obtain an Apple APNS Certificate
Release: 8.6 RU2 or greater
Open a new Broadcom Support case and request to have your certificate signed by Broadcom.
Attach the exported CSR to the case. Support will arrange to have the CSR signed and returned to you.
Modern Device Management
https://techdocs.broadcom.com/us/en/symantec-security-software/endpoint-security-and-management/it-management-suite/ITMS/Administration/About-Modern-Device-Management.html
Setting up MDM for macOS
https://techdocs.broadcom.com/us/en/symantec-security-software/endpoint-security-and-management/it-management-suite/ITMS/Administration/About-Modern-Device-Management/Setting-up-MDM-for-macOS.html