I'd say you need to worry about derailment but that would be something happening, and we all know that nothing ever happens.
In alls seriousness, I lived a few hundred yards from a train track in college and after about a week or so, I didn't even notice the noise anymore.