Even the OpenSSH people will tell you not to copy private keys around. Non-ephemeral private keys should be generated where they will be used. You can copy the public key...
Yep, there is no point in generating a new key if you can't get the other end to trust it. You could generate a new trusted key every time you log on (and encrypt with password), and invalidate the old key, making sure to copy it to your USB. That would roll the keys, and cause you to be locked out with that key if anyone used it. But a bit involved.
It doesn't take that long to arrive at the conclusion that you should not expose any secret to untrusted hardware.
So creating a one time use key for that computer is probably a good idea, you can revoke it once you are done using it and then it won't cause you any problems in the future.