I hate hurting feelings too, like I've always had a hard time breaking up with the wrong guys. I knew they were wrong way before I broke up with them.
Not that it matters now that I've found a perfect one for me.
But after I had a stalker, I learned super quickly that for stuff like that, when you're not with em and don't want to be, it's best to just say no, flat out, from the get go. I actually think it hurts less feelings in the long run.