Right! His "invisible hand" was never intended as a blanket justification for unregulated capitalism or selfishness! It was a loose metaphor used to argue against protectionism, and is nowhere presented in his works as a general theory of markets.
Indeed, it was largely ignored for nearly 200 years until it was popularized - and, I would argue, deliberately misrepresented - by Chicago School economists and other free market ideologues.