Two reasons:
1) If the port does get connected to another switch (and creates a loop), STP won't know until it receives a BPDU on that port. Since BPDU's are sent every 2 seconds, you'll have a window of vulnerability where a broadcast storm could ocurr.
2) Rapid spanning tree needs to know what ports are connected to other RSTP switches so it can negotiate with the switch at the other end of the link. If a port has portfast configured, then it assumes that there is no switch out that port and no RSTP negotiation will ocurr.