I think you probably need a fresh start, you probably have years of cruft in the background. I was like that, upgrading and transferring macOS from one machine to the next (starting with a Core2Duo MacBook Pro).
Finally started fresh on my M2 Mini, and again on the M4 Mini.
My M1 Air is still on the crufty install, and it's diabolical.