SSL certificate in PKCS#7 (.p7b) format cannot be installed completely by Symantec Messaging Gateway (SMG).
The PKCS#7 certificate is not compatible with SMG, and the SMG cannot convert it on its own.
Certificates must be in x509/PEM format in order to be installed into the Messaging Gateway.
openssl pkcs7 -print_certs -in certificate.p7b -out certificate.pem, with certificate.p7b replaced by the name of the existing cert, and your name for the exported cert replacing certificate.pem (ex. converted_smg_cert.pem).
For more information, please refer to the "Setting up Certificates" chapter in the SMG Administration Guide.