I put my SSH keys inside KeepassXC (regular Keepass supports this via plugin). Way better encryption and it automatically manages the adding of keys to the ssh-agent.

All my other SSH keys I don't have in there are plaintext on the disk. The ssh askpass is in userspace and easily spoofed, any local attacker could easily fish it out anyway. Full disk encryption at rest ought to be enough for most people.