You are showing low self-efficacy in your ability to take hits, or effectively prevent them connecting. This is a personal belief and not neccesarily fact. we all believe in our strengths and weaknesses, yet they may or may not be what we see them as. Maybe you do not trust your sparring partners.
The best thing to do is know that you have the training, the ability and the means to effectively stop an attacker.
perhaps too much time on tit for tat fight techniques and not enough of putting a stop to the advancing opponent. change your tactics a bit, maybe pussy footing around isnt your thing and your current fear is from fear over an extended time of your ability.
I too do not like my odds over time, my endurance sux, and I too dislike the gloves, so I looked for methods to quicken the end of the fight in my favour.
You are psyching yourself out. If you know the stuff, have demonstrated talent, then the barrier is internal belief. If practice isnt fixing it, maybe its time to find another style that better suits your mindset.
I am no expert, but play around with whats on offer.