You are setting up TLS encryption for an SMTP Prevent server and need to import the public key from the downstream MTA. When issuing the keytool commands as described in the documentation, you receive a Java exception showing that the input file is not an X.509 certificate.
Review the file you are attempting to import. If it contains a BEGIN CERTIFICATE and END CERTIFICATE line near the top and bottom of the file, it is most likely in PEM format, and should be converted to the binary DER format for use with keytool.
You can use OpenSSL (installed by default on Linux servers, but not on Windows) to check that the certificate is valid, while it is in the PEM format. Substitute the appropriate path and filename for the examples below:
openssl x509 -in mycert.txt -text
Examine the output to be sure that the certificate shows a public key signed by the correct authority. If the certificate is valid and intact, OpenSSL can also convert the certificate to DER format using the following command (again, substitute path and filename where applicable):
openssl x509 -in mycert.txt -inform PEM -out mycert.cer -outform DER
You can then continue with the import commands as shown in the MTA Integration Guide.