When you put a load on your solar cells the voltage will drop somewhat, this is normal. The more current you demand from the cells, the larger the drop will be.
10.4 volts is way too large a drop, you’re pulling more current than the cells are able to cope with. What capacity are your cells? (Open circuit voltage / short circuit current / power rating?) How much h current does the sunny buddy pull if you attach it to a 12 volt power supply?
I’ve connected Sunny Buddy to 12V adapter (measured 12.36V). After connection on input jack is still 12.36V, and on output jack (battery and load) is 4V. From that, I conclude that everything is alright with Sunny Buddy, it’s not faulty.
From that above, can you please give me new advice if you know, what can I do with voltage drop when I connect PV cells?
The current produced by PV cells is roughly proportional to the illumination(*), so to get the full expected power out of the cells, you need to be in full sunlight.
Please measure and report the short circuit current of your panel under the illumination conditions of your posted tests.
(*) The open circuit voltage does not depend strongly on illumination, so it is best to think of a PV cell as a current source in parallel with a diode.