IIRC, swap should be the amount of RAM you have x2 up to a maximum of 512 MB?

This -- although I'm unsure of the limit -- was true of 2.2 kernels. 2.4 will run without swap. But whether it runs faster or slower without swap is a matter of contention: it's probably an issue only the kernel team and/or some experimentation can settle (and may be determined by what you're actually running, too).