Have you compared your layout to Sparkfun’s Si4703 board? I would look very closely at you power supply decoupling around the Si4731, and even more closely at the grounding for the Si4731 (e.g. are you using the part with the exposed pad and is it grounded?, are you using a good ground plane underneath the IC?).
You can also take a look at SI Labs [AN383.](https://www.silabs.com/documents/public/application-notes/AN383.pdf)