Have you looked (carefully) into your thryoid? Blood tests won't always show if there's a problem. Constantly having a cold body temperature is a red flag, your waking body temperature is the most telling. I'm on T replacement at age 29, and I started going downhill around your age. I still don't feel anywhere near the way I did in my early twenties, and am looking into thyroid treatment. (TSH in normal range, but not optimal)
Anyway, your doctors are right to be cautious, Androgel or shots will treat symptoms, but the question is why are you so low? You have to know trt is likely to be permanent not temporary. For environmental reasons men today do tend to have lower testosterone than they did years ago, and there can be genetic reasons but that wouldn't explain it if the energy loss has been rather sudden. If you do go the trt route, try to get some weekly hcg injections to keep the production you do have up.