Yeah, although as you write out the logic inside the FPGA, it's annoying how fast cycles creep into the data path. A memory cycle here, a couple cycles for synchronization there, oh no this combinatorial path is too long so it can only run at 40MHz now unless I pipeline it with another register and now the path is another cycle longer, etc.
That's where my lack of confidence comes from atm, I don't know how many cycles things will have to be :/