you did the right thing bro, he can't prolly sustain much physical punishment if he's as bad off as you say, he's just lucky that you used the newspaper. I think that I'd to have put a foot in his ass. You can warn him though, next time it's a beatin, don't ever let me hear you talk like that to MY mom again, you do and it's your ass.
I understand he's in pain, prolly pissed at the world, but he chooses to put alcohol into the mix, if I folowed your post, he's a decent guy until he gets to drinking. That part is under his control....
Then you should have kicked your sister dead in her ass and told her that the next time she has an oppinion on how you handle your brother when he's disrespecting your mom, she can be next. Her choice....