On the face of it, it doesn't seem terrible if all the racists go and live in one place and leave the rest of us alone.
I'd guess however, that such people can't really live in their bubble and be content, especially when it turns out that life with only white people is really nothing special or exciting.
They'd need to keep the rhetoric going and demonise the people living outside their borders, making the whole area unsafe.