What you're talking about has been disproven by history time and time again. Social consequences are a huge part of how we suck less.
Making certain beliefs socially unacceptable is how we curtail their influence, not acting is if they're simply a difference of opinion.
And yeah, I have intolerant family members. For instance, my dad is a Holocaust denier. We don't talk.