I’m very new in AVR programming and just wondering is it possible to use bootloader UART (RX,TX) on Atmega8 while the main application use the same UART for communication with GSM modem.My fear here is about electrical circuit and problems that may come using UART in such a way.
Is there a bootloader that may work on port ohter than UART (is this possible???)
A bootloader just needs a way of selecting it and a way of getting data to it. Most bootloaders look at either a specific state on a pin just after reset or for a specific character on the UART to decide whether you want to enter the bootloader or branch normally to the main program.
As for getting data to the bootloader once you have entered it, you can use any means you wish - the UART, a bit-banged UART, I2C, SPI, or any other way of sending the byte stream.
In my latest application, I am using the UART and a bootloader to allow updating an ATMega3250, and letting it program the slave ATMega8 thru SPI (though I implemented ISP for programming the 8 rather than depending on a bootloader there).