It seems to me like the memory access itself is the main bottleneck either way, so it might not make much difference to speed but does free up a CPU core to be doing some other work.
But of course you might not have any other useful work to do, which is what I meant in my opening about it potentially not being worth it.