You are really never too old to start. It depends on what your priorities are. I mean if you are starting to learn self defense and exercise. The sport aspect is somewhat age limited. So, have fun at it and enjoy yourself.
I would suggest a type of Jujutsu, Aikido, Aiki jujitsu. Also tradtionally karate is supposed to be a defensive art. Unlike what most sports oriented schools teach. I do suggest checking out the schools in your area. Ask about affiliations. Check out their affiliations on the web. And even...
I think that there is a very good school in that area. Master Brian Frost runs a Koei kan karate school there. It is a Japanese style with a traditional base and a good real world self defense training style. He is a really good teacher. I am not sure about the actual address. :)
Frankly, as a black belt, it sounds like this person is a very immature black belt and really does not have an idea what the art is all about. If he did, he would not go around challenging you. Karate is for self defense,period! He should know that.
Being a karate practitioner for 24 years and a Jujitsu practitioner for 15 years I would lean towards a combination of the two. Perhaps even Hapkido. Another very effective and practical art is Bando (Burmese kickboxing). Also called Thaing. Karate is only impractical if you have a closed mind...