Interesting: my piano setup at home (Kronos→USB→Pianoteq→JACK→Firewire→Focusrite) seems to have a latency spike when I haven't been using it for a few hours. The first key I press has notable latency (~150ms?), after that it's normal. I wonder what in the stack causes that.
I know, I know. I use it as a synth too, and originally for piano, but then I discovered Pianoteq which has taken over the piano synth role. I still use it for other instruments :)