Manually setting them on the command-line like that gets the same behavior as doing it via the .ssh/config file.
And as another odd datapoint, if I do all the same from another FreeBSD machine I have access to (14.2p1) it works via IP address both ways (whether .ssh/config or all stacked on the command-line).
Is there a way to test "don't try and mess with QOS" to see if that's the issue?