- I'm afraid this is one of those nasty puzzles where I don't know the answer. Worst case: the guy who confidently told me SmallGroup(32,43) has this surprising property was wrong, or it's not the group of affine transformations we think it is.
It's true that in a semidirect product A ⋉ B with both A and B abelian, conjugation by any element leaves the first component in (a,b) ∈ A ⋉ B unchanged. (Another fun example: the rotation/translation group of the plane.)
The only way out I see is what Omar Antolín (npub1ums…7ra6) suggests: there's an automorphism f that multiplies b by 3,5,or 7 in a manner that depends nontrivially on a:
f(a,b) = (a, n(a)b)
where n(a) = 3,5, or 7.
Thanks for tackling this!