It exist within the CA Identity Portal Guide an incorrect statement regarding custom message in "Form Handler Examples"
https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-portal/14-4/administrating/identity-portal-administration/elements/backend/create-forms.html
Where the Validate Handler example refers to :
prop.error = "Invalid object name: " + prop.value;
However, error messages are handled within an array through prop.errors as documented within the Hotel Reservation Form Example
https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-portal/14-4/programming/developer-guide/form-handlers/hotel-reservation-form-example.html
prop.errors = [result.returnValue];
So custom message can be set with :
prop.errors.push("Here is custom error message from Validate Handler");
or
prop.errors = ['Here is custom error message from Validate Handler'];