I put a non specific answer for you in hardware.
Linux choices:out of the box $$ solution, redhat enterprise.
do it yourself:slackware would be my first choice as a base to get started, however you do it, your going to have to do some work to get it setup.
kernel compile with SMP and x64 support, not hard, but if youve never done it you might not be very sure of your work
install whatever apps you need like apache,mySQL, whatever you need
configure those apps
configure networking
configure samba if sharing with windows machines and doing direct filesharing
configure users and passwds
test everything and make sure it works
write some scripts for maintenance issues, linux has some apps like logrotate to keep system logs up to date
This isnt a job for rookies, it can be done by a rookie, but when your done you will know an enormous amount of info and it will take awhile.Ive been using linux for years, depending on what exactly this thing will do and how its needs setup,it would take me anywhere from 4-12hrs roughly to get going.If you have special needs for this setup you can add more time.Less if its just for filesharing only, just depends on how much storage and users are involved.
I dont know anything about the windows version of doing the same thing other than general ideas that are in both systems.
Either choice also means if security is an issue, doing some searching and installing all the current patches and close all the loopholes in either system which you can plan on taking a couple days of working on it off and on.