They are afraid for their property value, they are afraid of a population might need a higher level of taxation. Conservatives would build, but only villas that would not be affordable for ordinary workers.
Few local politicians want to invest in social housing. They prefer that the poor live in neighbouring municipalities.