Check your masquerade name in the Virtual SMTP Server (Properties - Delivery -Advanced) to ensure you have a match between PTR record for your IP and who you are saying HELO as. In addition, from the mail server itself, can you try a regular telnet session using port 25 and trying helo
and see if the session continues where you can enter "mail from:" and "rcpt to:". There is a great tutorial on how to manually test email connections from here: http://postmaster.aol.com/tools/telnet.html