Remote Monitor Service (RMS) fails to install on Site Servers if the Monitor Agent is already installed.
This is by design. RMS povides all the functionality of the Monitor Agent plus Agentless monitoring functionality so it would be redundant to have both RMS and the Monitor Agent installed. Unfortunately the installation of RMS does not automatically uninstall Monitor Agent.
Uninstall the Monitor Agent from the Site Server before installing RMS. This can be done by cloning the Altiris Monitor Agent Uninstall Policy (found at Settings>All Settings>Agent/Plug-ins>Monitoring and Alerting>Monitor>Windows>Rollout>Monitor Plug-in for Windows x64 - Uninstall) and then deleting the default target and creating a target for the Site Server. Additionally the Site Server may have to be excluded from the targets of any Monitor Plug-in Install policies, although the default target for these policies will automatically exclude any Site Servers targeted for RMS installation.