Hmm, like you say, that could happen but is unlikely unless I'm running some valuable stuff. The only thing network orientated in our router is the network.
As for not using the Windows Firewall, I just don't see the point. We run 3 computers (used to be four), all connected to the router and none of them have EVER got a virus, trojan, worm, etc. Sure we get spyware, but none of it has been malicious, usually just MSN cookies and the like, easily removable.