someone correct me if I am wrong.

At the kernel level, the xfs stuff is just something that patch the kernel such that the XFS module(if one choose to use it) can work properly. So patching it or not doesn't really matter if one does not have the intention of really using XFS. In that sense, just add 'xfs' to the extraversion keyword in Makefile of the kernel source(after make config) and everything should be fine as the compiled kernel would say 2.4.20-xfs when queried.