As a refinement, since you are satisfied with vesa as a universal driver after the framebuffer phase,
I suggest making the earlier phase just a scan to select a fb appropriate to the video hardware.
I've seen this in Ubuntu 10.04. Seems like one size doesn't fit all in frame buffers in Ubuntu, anyway.
In Ubuntu, one of the options is to suffer thru the framebuffer phase which doesn't last too long;
the subsequent X-session phase proceeds ok, then a brief fb phase on shut-down. I spent a month
on this because the initial Ubuntu 10.04 opted to not include SIS video's fb, which it turns out can
be made to work just as well as what those from what they call 'the big three' video suppliers.
Intel Core i7-4790K SR219 4.00GHZ 4Th Gen Used Desktop Processor CPU FCLGA1150
$64.99
AMD Ryzen 9 7900X3D Gaming Processor - 12 Core And 24 Threads
$449.00
Intel - Core i5-14600K 14th Gen 14-Core 20-Thread - 4.0GHz (5.3GHz Turbo) Soc...
$305.99
Intel Xeon E5-2697A V4 2.6GHz CPU Processor 16-Core Socket LGA2011 SR2K1
$37.99
Intel Core i5-8500 Six Core Desktop PC CPU Processor 3.00GHz LGA1151 SR3XE
$43.99
Intel Core i7 CPU Processors, I7 2600, i7 3770, i7 4770, i7 4790
$28.99
Intel Core i7-7700 Processor CPU SR338 3.6GHz 8MB LGA1151
$64.97
Intel Core i9-12900K Processor (5.2 GHz, 16 Cores, FCLGA1700) * Open Box * USA
$275.89
Intel Core i7-14700K - 14th Gen 20-Core (8P+12E) LGA 1700 125W CPU Processor
$373.96
Intel Core i7-13700KF - 13th Gen Raptor Lake 16-Core (8P+8E) Desktop CPU Proc...
$289.99