Ah, yes! Haha, I was really stupid and was only thinking of a totally clean/windowless desktop. Good thing you reminded me of this very obvious way an accidental desktop scroll could happen :)
Also, you're right, the same, opposite thing should undo the action, and unfortunately scrolling in Activities does change workspaces, so that would have to change too. But that is a very logical thing, so… Hmm, I guess the people are wise, and I was wrong… :/