Just wanted to update this in case someone else has the same problem. There was an issue with the SPI pins being reversed in the FPGA softcore. This was fixed by Alorium Tech, so if you have this problem make sure in Arduino you update XLR8BuildTemplate, XLR8Core and the Alorium Family boards. And thank you to Alorium for getting this fixed.