As a refinement, since you are satisfied with vesa as a universal driver after the framebuffer phase,
I suggest making the earlier phase just a scan to select a fb appropriate to the video hardware.
I've seen this in Ubuntu 10.04. Seems like one size doesn't fit all in frame buffers in Ubuntu, anyway.
In Ubuntu, one of the options is to suffer thru the framebuffer phase which doesn't last too long;
the subsequent X-session phase proceeds ok, then a brief fb phase on shut-down. I spent a month
on this because the initial Ubuntu 10.04 opted to not include SIS video's fb, which it turns out can
be made to work just as well as what those from what they call 'the big three' video suppliers.
Vintage DEC/CLEARPOINT INC. QED1 4MB Memory PDP
$149.99
VINTAGE DEC DIGITAL DISK FOR RK 05 HARD DRIVE
$89.99
VINTAGE DEC Digital RK11-D STATUS M7254 PDP 11 (B1)
$129.99
Vintage Harris S550 Front Control Panel Console w/ Key, DEC PDP
$2379.15
Vintage DEC Digital Professional 350 Desktop Computer PC350-D2
$1699.95
Vintage DIGITAL DEC MICROVAX II - 630QZ-A2
$2000.00
Ultra RARE Orange Screen DEC VT320 Powers On Digital Vintage Terminal Monitor
$195.00
VINTAGE Green Screen Digital Terminal Monitor Model DEC VT320 - Powers On
$195.00
Vintage DEC/CLEARPOINT INC. -QRAM-2 SAB-1 1MB Q-BUS MEMORY MSV11-QA (B2)
$129.99
Vintage DEC Digital Equipment PDP-11 Half Blank Front Panel 19''
$69.99