As a refinement, since you are satisfied with vesa as a universal driver after the framebuffer phase,
I suggest making the earlier phase just a scan to select a fb appropriate to the video hardware.
I've seen this in Ubuntu 10.04. Seems like one size doesn't fit all in frame buffers in Ubuntu, anyway.
In Ubuntu, one of the options is to suffer thru the framebuffer phase which doesn't last too long;
the subsequent X-session phase proceeds ok, then a brief fb phase on shut-down. I spent a month
on this because the initial Ubuntu 10.04 opted to not include SIS video's fb, which it turns out can
be made to work just as well as what those from what they call 'the big three' video suppliers.
Amiga 3000 Motherboard ReAmiga Rev. 1.3 and Daughterboard Rev 9
$119.00
Amiga 500 Computer, Power Supply, And Workbench 1.2, Tested
$390.00
Amiga 500 Mitsumi Keyboard Tested Working
$60.00
BlueSCSI V2 WiFi (Desktop) Modern Storage for Vintage Computers Latest Model
$53.50
Cyberstorm PPC Amiga 4000 Phase5 turbocard with 128 Mb RAM -for parts or repair
$1500.00
Amiga External Drive Adapter and LONG Cables. Works with Gotek and Amiga Drives
$32.00
$237.18
November 1985 Volume 3 No. 11 Family Computing Magazine, Commodore Amiga Preview
$19.50
AMIGA 2000 RGBTOHDMI ADAPTER RGB2HDMI DX HDMI
$83.33
AMIGA 500 COMPUTER COMMODORE REV 6A HEATSINKS IC PAL VER KS1.3 TESTED/WORKNG L24
$379.99