I know you say you have your ports forwarded but does your router have a firewall built in? if so it might detect the torrents as a DOS attack and temporarily block the IPs. Doesn't sound likely though.
If all you're talking about is occasional line drops then its prefectly normal for either your router or the exchange to drop the connection every now and then.