I believe that in 1Passwd you can define / preselect a key per host now. So you can pinpoint key -> host. Some hosts have firewall rules that will block after X attempts were X might be low.
However the agent still has access to all your keys, obviously.
Ah cool. I worked around by storing the public keys in my dot repo and use the identity file ssh config option for said host. Great if I don’t have to do this anymore.
Next level config madness: Use different ssh keys per GitHub org ;).
However the agent still has access to all your keys, obviously.