From
http://www-01.ibm.com/support/docview.wss?rs=463&context=SSKTMJ&context=SSKTWP&dc%C3%9B520&uid=swg21233872&loc=en_US&cs=utf-8&lang=en :
"Customers with 64-bit hardware and operating systems will be able to run either (or both in separate partitions) 32-bit or 64-bit Domino."
So, I assume that you can indeed stop the server, install 32-bit Domino and start the server again. As always, make a backup before you start this procedure.
So *I* don't see an issue, but that doesn't necessarily mean that there aren't any... If you have a test system (even a virtual one), you might try it there.