When applying a Web of Trust difference between a follow and a follower is critical. A Web of Trust based on followers will never work, at least not in a decentralized way, for the reasons you mentioned.
I like how you are thinking of it in terms of a fundamental source of scarcity. With a Web of Trust based on follows that scarcity starts with the user's follow list, everything else is derived from that input.
The nice thing when everything is derived from that 1 input is that any problem in the output can be solved with a simple 'Unfollow'.