Is it just a loud "whooosh" sound coming from the computer or do you hear a whining sound? If it's a whine, something is probably resonating. If you can then pinpoint it through the process of elimination, you should throttle down the fan(s) in question. If the mobo does not allow you do do it, you can cut the negative wire and place a small rheostat inline for speed control.
Now, if it's just regular fan noise, there's not a hell of a lot you can do about it, besides trying to find quieter fans. Opening the case will cool it quite a bit, but of course you'll have problems with dust. Blow it out with some compressed air once a week and you should be ok.
I've had the side panel removed from my media server for a year. There's just too much stuff in there for it to stay cool...6 hard drives, two optical drives, etc., cables all over the place, not to mention that it's overclocked into oblivion. I had so many fans that it sounded like a jet taking off. I took off the side panel and now I only have to run 6 fans instead of like 10.