I've been using this combination for a few days now, and love it. I found xfce4 to be very fast and light-weight, but I didn't much care for it's panel and lack of certain features. I was looking for something on google and happened to see that some people use gnome-panel within xfce4. The result is that you have gnome's functionality at your fingertips, but within a much lighter window manager.

I'm sure that it's not as fast as running pure xfce4, but it seems to be significantly faster than gnome (and kde). It's very easy to get rid of xfce's panel and taskbar and use gnome-panel using xfce4's session editor (which shows up in gnome-panel's notification area). If you want speed, but don't want to give up functionality this seems to be a really nice compromise between the two.