I've never actually started up X locally when remastering, I've always just started up VNC and then connected from a different computer

But I don't think that's what your trying to do here.