the people that it seems to matter to are those that are either hypocritical and judge others despite them having a stained or similar past, or to those that live by what they consider "high standards" that they expect their SO's to fit into. Or the prudish.
I have not been a perfect person in my past, nor am i in the present. So the past is something I really don't care about to judge a person. Similar to what NYCGirl said, who are they today and how do they make me feel about tomorrow, because that is the way I want to be judged. If they can't deal with it, then that's not the type of person I should be with anyways.