the issue if that you have your own server, sadly dozen of server are unmaintained, or hosted by morront mysoginistic ... so if you don't defederate them you are toast.
I have both the hardware and the knowledge to run my own server but i won't as i don't have time to do the defederation / moderation.
maybe ask Jerry 🦙💝🦙 (npub1vtx…e7wk) if he can (kindly) give you an easy to use list of server to defederate so they cannot annoy you anymore ?