Yes you should be concerned. The ability to restrict the data coming from the IPSEC tunnel endpoints should be an absolute prerequisite for turning up the tunnels. We do the same for any external VPN users(usually vendors), only allowing their VPN credentials to access the resources required for them to do their task. For example, and HVAC tech may be able to VPN in, but they can only open port 5900 for VNC to an internal HVAC system.