Blacks have every right to create private organizations that are exclusively black, it is the right of association. With that, since rights are individual and not collective, this same holds true to anyone who wishes to organize based on some criteria: race, belief, interests, sex, etc. It is the right of people to associate with others of common background.
What is wrong is to claim is that only certain groups have this right and others don't, since this infringes on the rights of others based on your personal opinion, and opinions are irrelevent. So, when people claim that all white organizations are racist, or all male groups are misogynistic, it is perfectly reasonable to counter this claim by demonstrating the acceptance of the opposite group.
Groups, based on race or gender, while legal, do nothing but promote "group mentality", which is not advantageous to the betterment of the individuals. It cements the idea that one's value and acheivements are only as great as the group that one belongs. Both white supremacist and black identity groups use these venues to act as victims.