Have you calculated the baud rate error for the baud rate you’ve chosen and the crystal’s frequency? Need to be less than about 1.5%.
Also, if you go to RS232 voltage levels, some of those level shifters have bandwidth limitations.
If the UART is interrupt driven, you can run in to limitations for devices that lack a hardware FIFO.
Generally speaking, UARTs at 115Kbps and up are arguably a misapplicaiton.