We have a problem with Robertsonics WAV Trigger and Tsunami Super WAV Trigger.
Problem: At random times, modifications are done on the SD card resulting in glitches in the wave files.
This occurs randomly (hours to months) and has happened both while developing and with installations is running untouched.
The symptom is noise, cracks, pops or distortions suddenly appearing at the output.
Extracting the SD card and examining the files reveal that they have in fact been changed and that the file now contains the glitches we can hear coming from the output.
This has happened with 1x project using WAV Trigger and 2x projects using Tsunami Super WAV Trigger.
The projects all use different SD cards, from different brands.
The projects are in very different environments (indoor, outdoor).
Power supplies have been different (#1 USB, #2 9V+LDO, #3 24V+5VDCDC+3.3VLDO)
Power checked to be perfectly clean on two of the three problematic installations
We have also had projects that successfully use the product and the only difference we know between the working and problematic installations is that the working ones use the 16 trigger inputs, but the projects that crunch SD cards use TX/RX communication with an external microcontroller.
This is an interesting issue I have not heard of before. It sounds like there might be something in the serial data that is causing the codec or Cortex M7 to modify the WAV files somehow. I presume the serial strings are always intended to be consistently identical. I suppose there could be some kind of electromagnetic interference going on that is causing this issue, however, it seems like the issue occurs in diverse environments with diverse power supplies.
Or perhaps the external microcontroller is randomly sending rogue serial data to the WAV trigger serial lines which is causing these issues. I would suggest that you try and test the serial output on the external processors and see if there is something being sent before the modifications on the SD card files happen. I think any kind of power cycling could do this, among other reasons. Is the WAV Trigger the only device on the serial lines?
What happens if you delete and replace the audio files?
Just to clarify, both the WAV Trigger and Tsunami are “read only” devices. They contain no code capable of writing data to the microSD card, so I doubt very much that this is a software/firmware issue. I communicated directly with one other person who experienced something similar, and I was able to confirm that the microSD card was indeed corrupted. If I recall, the problem was electrical, involving an electrically noisy environment (e.g. large relays) and the solution was to re-think the power/grounding strategy. I believe this solved their problem.
@TS-John: The serial communication is very basic and uses the Arduino Library provided by Robertsonics. Commands are sent infrequently and no other data (or devices) are present on the line. Communication has been checked with both scope and logic analyzer - it is stable and noise free and the devices have never missed a command. Power quality has also been checked and the noise is below 25mV (measured with 10MHz bandwidth). If I delete or replace the WAV files, everything works well again, for a time.
@Robertsonics: I understand these are read-only devices and that writing to an SD card is not really something that just happens due to a minor (Q)SPI communications glitch. But - the problem only occurs in files being actively played. Doing a RAW copy of the entire SD card shows that no other modifications are present. For our latest installation: Power quality at the Tsunami has been checked, nothing above 25mV over 10MHz. The digital subsystem (controller and Tsunami) are running off their own linear power supply with hefty bulk capacitance as well as smaller ceramic decoupling caps and all power is properly star routed. The only interference source of any magnitude is a WS2812 LED strip, but this is separately powered and not located near the Tsunami. The Class D 30W amplifier emits only minor (far below anything that should remotely cause issues) EMI primarily around 250kHz. No relays, no motors. The problem has has also occurred on the lab bench a few times with only Arduino UNO, level converter and Tsunami, USB powered. And then a few times during long term (days to months) operation - I think this was the sixth time now, twice on a WAV trigger and the rest on two different Tsunamis.
Frankly, I am at a loss as I’ve gone through checking everything I could think of.
The fact that you’re observing this on both the WAV Trigger and Tsunami seems further evidence to me that it’s not a logic or software issue. They use different processors and completely different firmware and hardware resources to interface to the microSD card. Also, the WAV Trigger audio engine, which includes the microSD card interface, has been used in a number of commercial products with no reports of card corruption that I’m aware of.
I’m certainly not trying to disregard your problem, and am sorry that this is happening, but I’m at a loss to explain it other than some electrical issue that is causing noise to the microSD card. Perhaps it is more prone to interference/noise during read operations, which would explain the corruption pattern.
I’ve run both the WAV Trigger and Tsunami many, many hours during development, and on several occasions have left one or the other continuously playing for weeks at a time. I have never had this happen or observed any corruption. It’s hard for me to troubleshoot something I’m not able to reproduce.
I think you implied above that you have used one or both products before without this issue. Do you have WAV Triggers and/or Tsunamis running that don’t ever have this problem in similar installations? If so, can you give me some idea of the number of units you have experience with beyond these 3 that seem to be producing corruption?
I’m wondering if perhaps there could be a manufacturing issue where the microSD socket pins are not soldered adequately and have intermittent connection. Just a shot in the dark…
@robertsonics I completely understand, and yes we have had Tsunamis and WAV Triggers running for months without any issue. There has been at least three other projects (probably more, before my time) using WAV Triggers or Tsunamis, reportedly without problems. Only the last three have issues, but otherwise have very little in common except for WS2812 LED strips, Arduino UNO and WAV Trigger/Tsunami being involved.
Shots in the dark are OK at this point. If I get a chance to have at it before the latest installation ships out again, I will take a close look at the soldering under a microscope, re-solder the SD card just to be sure and hook up wires to be able to check power stability directly at the SD card connector, as well as bring a proper scope and check for noise all the way up to 200MHz.
If not, I will get back to you if I ever find a way of reliably reproducing this.