Kernel stuff... interrupt controller might be it. There is also an occasional debug message finding its way to the console (i.e., the laptop screen) indicating that a WiFi interrupt timed out: "ath0 device timeout (txq 1, txintrperiod...)"; that's without X11 server.
I agree, I also wouldn't bet on the USB stack.
I'll give it a try with different i686 hardware, and with a AMD64 machine.
Then the mailing list.