I have my box running on a 2.4 GHz core 2 Quad, 2 gigs of RAM, and a 500GB sata drive. It's what I had lying around
It's a test bed to try out the technology, and I have just about every available package running on it, and it hosts email for 1 domain. Granted I'm one user, and I'm not using it heavily, but I can't imagine it not being able to comfortably handle 10-20 users.
With all server type stuff, even Microsoft, drive speed and RAM are much more important than CPU; for instance, I recently upgraded a 32 bit win 2003 machine with 2 gigs of ram to 16 gigs of ram and 64 bit Windows 2008, and despite 2008 being newer, the machine runs noticably quicker just due to the RAM.
Drive speed as I said is important too - for my server boxes that I build, the boot drive is an SSD drive, which it will use for swap as well, and the data storage drives are SATA. There is a big tendency in this industry to oversell what you need for hardware.
So load up on RAM because its cheap, and take the extra time to configure the 64 bit version, and you'll be happy.