This is the issue throughout all of QM that bothers me - the use of classical looking fields for quantum particles. How can you get even a rough idea of the energy levels of an atom with more than one electron when the electron whose energy you're computing is embedded in clouds of all the other electrons?
There are arguments like https://en.wikipedia.org/wiki/Born%E2%80%93Oppenheimer_approximation
More generally though I don't doubt the AB effect is real simply because AB is just probing one tiny part of a body of theory that works. But I can sympathise with someone worrying about this specific experiment.