Yes, you would use a hub or switch not a router though.
Or you could use a crossover patch lead and connect the ethernet port of the linux machine to the ethernet port of the windows machine directly
You'd then set ICS on the win pc and on the linux machine set the internet gatway to the IP of the win pc