Usually enabling EEM SDK Logging would give a better idea on where the loss of time is. Other than that, it maybe possible that improper network resolution of EEM/SDM servers might be the root cause as well. EEM server requires reverse PTR records for hostname resolution.
EEM server requires reverse PTR records for hostname resolution. This can also be done by putting the involved servers into each machines' host file.
To verify the same on each server perform an nslookup for both IP address and hostname. For example:
On EEM Server:
c:>nslookup SDMServerName <enter>
c:>nslookup IP_Address_of_SDMServerName <enter>
ON SDM Server:
c:>nslookup EEMServerName <enter>
c:>nslookup IP_Address_of_EEMServerName <enter>
If you have EEM configured for failover, then repeat the above process on each EEM Server Node, AND repeat the lookup against each EEM node from SDM Server too.
It's also recommended to try the above using the short name/netbios name AND Fully Qualified name for these servers just to see if there is a difference in the response.