Was your wife APPRECIATIVE of all the "work" that you did during the day?... or did she bitch about the tasks that you didnt get to? Did she regularly make comments like, "Damn dinner is late AGAIN? WTF were you doing all day? How come the laundry is still in the basket unfolded? Why cant you keep the kids quiet when I come home? I just stepped on a fucking lego and goddamn it hurts. Why arent the kids' toys picked up?"
It isnt that women use the work they do as an excuse to avoid intimacy. Their work is often unappreciated and even made fun of. Tell me, would you want to be intimate with a spouse that treated you this way? Not to mention the fact that though you BUSTED YOUR ASS - you were not the one who was physically drained by having to go through the pregnancies. Imagine being pregnant while taking care of your sub 2 year old. Think about how much work you would have been able to get done being sick, exhausted and stretched to the limit both physically and emotionally, forget about mentally as your mind would have been turned to mush.
Kudos to you for being a STANDUP father.

But in the end your children and your wife will be your greatest fans - because you WERE AND ARE SUCH A WONDERFUL DAD/LIFE PARTNER. The opinions of society will not change nor does that likely matter to you.