For everyone else coming here, in the end I solved it by using the Wav Trigger Pro along with a modified firmware (half clock speed, lower temps) + the Qwiic Amp - works perfectly along with other Qwiic devices.
Read more here: WAV Trigger Pro - runs a bit hot - #15 by michaelheiml82