yes, they were accused of that contemporaneously, as well.
Tzoreff's book (which had me crying in some parts and applauding in others) gives a good sense of why they had to fight it this hard - and contrast Reb Binyomin’s positions to those of eugenics-supporting Ruppin. (About whom Etan Bloom's Ph.D. thesis and eventual book are fascinating and horrible. LMK if you want a copy of the thesis.)