I think this question is what warps the perspective of rape apologists. When we rush to the defense of abusers (whether personal or famous) we're saying "I couldn't see this, so either I'm blind or it's untrue."
No. I believe it.
And with that belief my world is a colder and more monstrous place.
Gaiman stole a little bit of wonder out of my heart. He did far worse to his victims, but his crimes infest his body of work, and leave a stain on all of us he touched.