It looks like Domino tries to connect to the domain itself instead of lookup MX records for the domain. This happens when Domino is unable to resolve mx records
You did not mention the OS so I am assuming it is some Windows version.
Couple things to check:
1)if you have more then one NIC in your server, check that NIC with correct DNS settings is listed first in the binding order
Network connections-Advanced-Advanced settings, change the order so the active NIC with correct DNS settings is listed first
2) check notes.ini for any DNS settings and correct them if needed (machine DNS setting might change, but domino still using old values from notes.ini)
3) check registry if you have any entries in SYSTEM\CurrentControlSet\Services\TCPIP\Parameters\NameServer . remove the entries if they are there.
If you domino server was upgraded from 5 to whatever current version, the entries might be there with incorrect info