UPDATE
Since the RFID reader can only tell if a device got close, but does not tell if the device was removed, I changed my setup and use the Hall Effect sensor now, which gives both “in range” and “out of range” readings: SparkFun Mini Linear 3D Hall-Effect Sensor - TMAG5273 (Qwiic)
ORIGINAL
For everyone else coming here, with help from forum people we have managed to fix the RFID reader, and I built the solution based around it.
More to the RFID reader: Qwiic RFID wrong readings after 20 reads - #50 by michaelheiml82