RTK Reference Station - Boot Loop

I’ve been working to setup a RTK Reference Station / RTK Facet configuration communicating over SiK Telemetry Radios. While trying to debug the radio communication, the Reference Station has now gone into a boot loop with the following error:

ELF file SHA256: 0000000000000000

Rebooting...
ets Jul 29 2019 12:21:46

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1240
load:0x40078000,len:13012
load:0x40080400,len:3648
entry 0x400805f8
[  mum    2-hal-cpu.c:211] setCpuFrequencyMhz(): PLL: 480 / 2 = 240 Mhz, APB: 80000000 Hz
[    27][D][esp32-hal-adc.c:210] __analogReadMilliVolts(): eFuse Vref: Supported
[    28][I][esp32-hal-adc.c:240] __analogReadMilliVolts(): ADC1: Characterized using eFuse Vref: 1065

[    33][D][Begin.ino:56] identifyBoard(): Board ADC ID (mV): 1133
0x3d - SSD1306 (64x48) OLED Driver
0x60 - Crypto Coprocessor
Display started
LittleFS Started
[   386][E][vfs_api.cpp:102] open(): /littlefs/configureViaEthernet.txt does not exist, no permits for creation
ZED-F9P firmware: HPG 1.32
SparkFun RTK Reference Station d4.1
[   479][D][NVM.ino:1670] loadProfileNumber(): Using profile #0
Reset reason: ESP_RST_PANIC
Initializing microSD - using SDIO, SD_MMC and File
microSD: Online
[  2346][D][NVM.ino:124] recordSystemSettingsToFileSD(): Removing from SD: /SFE_Reference_Station_Settings_0.txt
[  2410][D][NVM.ino:142] recordSystemSettingsToFileSD(): Settings recorded to SD: /SFE_Reference_Station_Settings_0.txt
[  2426][D][NVM.ino:172] recordSystemSettingsToFileLFS(): Removing LittleFS: /SFE_Reference_Station_Settings_0.txt
[  2594][D][NVM.ino:184] recordSystemSettingsToFileLFS(): Settings recorded to LittleFS: /SFE_Reference_Station_Settings_0.txt
[  3344][E][vfs_api.cpp:102] open(): /littlefs/SFE_Reference_Station_Settings_1.txt does not exist, no permits for creation
[  3346][D][NVM.ino:545] loadSystemSettingsFromFileSD(): File /SFE_Reference_Station_Settings_1.txt not found
[  3356][E][vfs_api.cpp:102] open(): /littlefs/SFE_Reference_Station_Settings_2.txt does not exist, no permits for creation
[  3366][D][NVM.ino:545] loadSystemSettingsFromFileSD(): File /SFE_Reference_Station_Settings_2.txt not found
[  3376][E][vfs_api.cpp:102] open(): /littlefs/SFE_Reference_Station_Settings_3.txt does not exist, no permits for creation
[  3386][D][NVM.ino:545] loadSystemSettingsFromFileSD(): File /SFE_Reference_Station_Settings_3.txt not found
[  3397][E][vfs_api.cpp:102] open(): /littlefs/SFE_Reference_Station_Settings_4.txt does not exist, no permits for creation
[  3407][D][NVM.ino:545] loadSystemSettingsFromFileSD(): File /SFE_Reference_Station_Settings_4.txt not found
[  3417][E][vfs_api.cpp:102] open(): /littlefs/SFE_Reference_Station_Settings_5.txt does not exist, no permits for creation
[  3427][D][NVM.ino:545] loadSystemSettingsFromFileSD(): File /SFE_Reference_Station_Settings_5.txt not found
[  3438][E][vfs_api.cpp:102] open(): /littlefs/SFE_Reference_Station_Settings_6.txt does not exist, no permits for creation
[  3448][D][NVM.ino:545] loadSystemSettingsFromFileSD(): File /SFE_Reference_Station_Settings_6.txt not found
[  3458][E][vfs_api.cpp:102] open(): /littlefs/SFE_Reference_Station_Settings_7.txt does not exist, no permits for creation
[  3468][D][NVM.ino:545] loadSystemSettingsFromFileSD(): File /SFE_Reference_Station_Settings_7.txt not found
Profile 'Profile1' loaded
GNSS configuration complete
No GNSS date/time available for system RTC.
[  5169][D][RTK_Surveyor.ino:973] setup(): Boot time: 5169
Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.

Core  1 register dump:
PC      : 0x4010873f  PS      : 0x00060b30  A0      : 0x80108a7c  A1      : 0x3ffd1390
A2      : 0x00000000  A3      : 0x0000003d  A4      : 0x00000000  A5      : 0x0000002e
A6      : 0x00000002  A7      : 0x00000000  A8      : 0x00000001  A9      : 0x3ffd1410
A10     : 0x00000000  A11     : 0x0000002e  A12     : 0x00000002  A13     : 0x3ffc7c5c
A14     : 0x3ffbd288  A15     : 0x00000000  SAR     : 0x0000000a  EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000034  LBEG    : 0x400922c0  LEND    : 0x400922cb  LCOUNT  : 0x00000000


Backtrace:0x4010873c:0x3ffd13900x40108a79:0x3ffd13d0 0x4010758d:0x3ffd1400 0x400e7512:0x3ffd1430 0x400f81b8:0x3ffd14b0 0x40102d86:0x3ffd14e0 0x4013accd:0x3ffd1500

Appreciate any assistance you can provide.

What is your wiring setup for the radio?

We had been trying to output data to the radio via the TX2 on the terminal block and the radio was powered from a separate power supply.

At this point, though, there is no radio connected and the RTK Reference Station is powered via USB-C.

Okay, what was the wiring setup when it was hooked up? Like which pins to which ports, what voltages, was the external supply grounded to the reference station, etc…

If you attached the radio, then the ref station started misbehaving, then you removed the radio, and the ref station is continuing to misbehave, it indicates that the wiring setup may have damaged the ref station.

Some troubleshooting steps (I’d check to see if it boots correctly after each step):

  1. Disconnect all peripherals, including SD card and antenna

  2. Re-flash system firmware (v4.0 is the latest release)

  3. Re-flash ZED F9P receiver firmware (v1.32 HPG is the latest release)

  4. Perform a factory reset

These will isolate whether this is a software/firmware issue, or if there is some hardware damage.

The radio was not attached around the time the Reference Station started misbehaving, when it was connected previously the radio was receiving 5V and ground from an external power supply and the radio RX was connected to the Reference Station TX2 (the radio was showing a solid green and blinking red LED).

Regarding the troubleshooting steps:

  1. Still booted incorrectly

  2. Still booted incorrectly

  3. Error during FW update:

  0.0   - Opening and buffering image file
  0.0   - Verifying image
  0.0   - image 0 size 728620
  0.0   - image 1 size 593056
  0.0   - config size 0
  0.0   - Trying to open port \\.\COM8
  0.0   - Setting baudrate to 9600
  0.2   - Sending training sequence
  0.3   - Received Version information
  0.3   - Receiver currently running SW 'EXT CORE 1.00 (0fa0ae)'
  0.3   - Receiver HW '00190000', Generation 9.0
  0.3   - Sending ROM CRC Poll
  0.3 ROM CRC: 0x118B2060
  0.3 u-blox9 ROM1.02 hardware detected (0x118B2060)
  0.3  Getting Port connection to receiver
  0.4   - Connected port is: USB
  0.4  Starting LDR TSK
  0.5   - LDR TSK started successfully
  0.5  Identify flash loader
  0.5   - Uploader version 5.1 detected
  0.5  Stop GPS operation
  0.5  Detecting Flash manufacturer and device IDs
  0.7   - Flash ManId: 0x009D DevId: 0x6015
  0.8   - Not merging anything
  0.8   - Flash size:   2097152
  0.8   -  Flash block:  512 x 4096
  1.0   - Setting baudrate to 115200
  1.2  Receiver info collected, downloading to flash...
 10.8   - WriteFile err=31
 10.8 ERROR: SendWrite failed.
 10.8 Firmware Update FAILED
  1. The Reference Station will now boot, but gets stuck in Base mode and doesn’t seem to allow the mode to be switched:
rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1240
load:0x40078000,len:13012
load:0x40080400,len:3648
entry 0x400805f8
0x3d - SSD1306 (64x48) OLED Driver
0x60 - Crypto Coprocessor
Display started
LittleFS Started
SparkFun RTK Reference Station v4.0
Initializing microSD - using SDIO, SD_MMC and File
microSD: Online
Profile 'Profile1' loaded
Module failed config block 0
Module failed config block 1
Module failed config block 2
GNSS configuration complete
No GNSS date/time available for system RTC.
Bluetooth SPP broadcasting as: Reference Station Base-845E
Base config fail
No GNSS date/time available for system RTC.
SD card size: 29.1 GB / Free space: 27.5 GB

SparkFun RTK Reference Station v4.0
** Bluetooth SPP broadcasting as: Reference Station Base-845E **

@swells Any further guidance you can provide? I’m trying to determine if this is something we need to have RMA’d. Appreciate the assistance.

Hey wrp,

My apologies, never got a notification when you responded. After discussing internally, we think an RMA is the best course of action if you are interested; this doesn’t sound like any of the common failure modes we see. Can you please PM me your order number from the Reference Station purchase so I can have someone reach out to you?

@swells Thanks for taking care of the RMA. Following-up on the SIK Telemetry Radio connection to the RTK Reference Station – is there any documentation you can point us to for the wiring between the two? I want to ensure we’re setting everything up properly.

For using the Reference Station and SiK V3 Radios, we don’t have any specific documentation (all RTK devices in the Reference Station form factor are made with a “bring you own radio” philosophy, as the coverage range users need is extremely variable in a fixed base setup).

In the below tutorial, we walk through setting up a temporary base station using a raw ZED-F9P breakout board, including attaching the SiK radios to it; discusses some of the pitfalls, correct wiring, etc… If you search the page for “What about fast update rates?”, it will take you to the radio section:

https://learn.sparkfun.com/tutorials/se … orary-base

I’d suggest, especially after reading the tutorial above, that you look at adding a TTL level shifter between the radio and the Reference Station if you have trouble with radio stability or range of reception:

https://www.sparkfun.com/products/19626

Link to the hookup guide for the Reference Station, under the heading of “I/O Screw Terminals” you’ll find specifications for each terminal input (including import notes about using RX2 and TX2 for radio hookup):

https://learn.sparkfun.com/tutorials/sp … okup-guide

Link to the Sik Radio datasheet (bit sparse, but helpful for device capabilities):

https://cdn.sparkfun.com/assets/3/9/8/c … _Guide.pdf

Apologies for throwing the kitchen sink at you for documentation, but we generally choose tutorials to build based on what questions a majority of users ask us about, and many users seem to buy these with the intent of using them as fixed base NTRIP servers, so we don’t run into questions about radio hookup very often.