"My least favourite version is when they give the definition for 0 and say that it's 'by convention'"
Yes. I remember when I was a university student, I was upset a lecturer said that 0!=1 by convention.
Of course it isn't by convention. There is exactly one permutation of the empty list, namely the identity.
If you want n! to count the number of permutations of a list of length n, then there is no other option but to have 0!=1.