I have seen this phenomena on an MSP430. All the power needed to run the processor was available as parasitic power from (in my case digital) input pins. That had me completely puzzled for days “But there is no #$%@# power! How is this thing running?”.
Of course this is not a good way to power a chip! It really needs its power pins connected.
Now your question is likely: “Can I ever get the chip to power off?”. The answer is yes if you can get the analog voltage input to go to zero as well. It may also be possible to short the Vcc to ground on the Arduino when off, although I have not tested this nor know of any possible complications in so doing.
This is normal and expected behaviour. The I/O pins of almost all microcontrollers are protected by “back to back” diodes that prevent the voltage on the pins going too far from the power supply rail voltages. This gives some ESD and spike protection, although this is limited due to the size of the diodes. Normally the only exceptions are when a pin serves double-duty as a high-voltage programming pin.
As Skye says, you need to find some way to keep the voltage on that pin low when the microcontroller is powered off.