Most servers I connect to have the option PasswordAuthentication set to no, meaning I do more often than not see an error:

Permission denied (publickey).

The reasons for this are multiple, but in my scenario, this happens because there are no identities (keys) present in the SSH agent.

SSH agent

SSH agent holds OpenSSH keys used for public key authentication (public key authentication method is vastly superior to plain-text password authentication and should be preferred whenever possible), and hands the right one over to the server during connection. OpenSSH keys in itself is a rather broad term. For sake of this article, the term OpenSSH key here refers to the context of public key authentication. Public key authentication actually requires a key pair - obviously the public key and it's matched counterpart, a private key.

The default SSH agent shipped with OpenSSH is ssh-agent. It has a rather basic feature set, as it expected. There are software packages available that can acts as an SSH agent and they feature sets differ. There can be only one active SSH agent running on the system at a given time.

Keyrings and Password managers

Many distributions ship with Gnome Keyring, a password manager/keyring (branded also as User Credentials Manager, implying it is able to handle multiple types of credentials). These two terms are not the same, but for clarity, this article individually refers to Gnome Keyring as keyring and to KeePassXC as a password manager. Both software packages overlap greatly in the features primarily discussed here, thus the terms appear together.

Both KeePassXC and especially Gnome Keyring can handle multiple types of secrets or user credentials (some of them overlapping) such as passwords, security certificates and Freesdesktop.org secrets in addition to the keys such as GnuPG keys, and the core of this topic, OpenSSH keys. Keyring's/password manager's ability to handle OpenSSH keys means it also features an SSH agent implementation.

Private key passphrase

Private key can and (unless there is a valid reason not to) should be additionally protected with a passphrase. If protected, a valid passphrase is required before the key can be used for an actual authentication.

Extending the previous definition, Keyring's/password manager's ability to handle OpenSSH key, additionally means, it can store, and later recall, the stored private key's passphrase.

Passphrase prompts

In the most basic sense, ssh-agent prompts for a passphrase when the passphrase protected key is added into it. This process is synchronous in a sense that the terminal expects user to provide the passphrase before next command can be issued. The agent is preventing subsequent prompts for the same key already added into it, until the key is finally removed from it. Reducing the amount of passphrase prompts is in fact the main responsibility of the agent.

Keyring and Gnome Keyring work differently compared to the ssh-agent with respect to adding keys into their respective agent implementations. When configured for this, user is not required to provide passphrases for individual keys, as the passphrases are stored within keyring/password manager's database. Instead, user is prompted for a single master password when unlocking the keyring (with Gnome keyring there usually is no visible prompt for the SSH passphrase at all. This is by design, as an user account password is used as a keyring master password, unlocking the keyring automatically after user logs in) or the database.

The problem arises in situations where the prompt is invoked asynchronously - when there is no terminal associated to it. This is generally the case with programs employing a graphical user interface (GUI). Since Gnome Keyring and KeePassXC have their own graphical interfaces, they are both affected.

SSH_ASKPASS and ssh-add

Manipulating keys in the agent is done by ssh-add. While there are currently multiple agents available, there's just a single widely used ssh-add. All OpenSSH agents mentioned strive to be compatible with the ssh-add command, otherwise they would not be very useful.

One particularly problematic scenario with the passphrase prompt and both discussed GUI programs is the -c parameter of ssh-add (while technically KeePassXC and Gnome Keyring would work well enough as an agents without implementing this functionality, they did implement it over time, because of the community requests for a full compatibility with the ssh-agent), flagging the key added to the agent for confirmation before being used. The key flagged like this is still added to the keyring/password manager's agent, but expects to prompt the passphrase into a floating dialog the moment a key is actually used. This functionality in turn requires at least a properly configured SSH_ASKPASS environmental variable, pointing to a working graphical prompt implementation.

KeePassXC and user confirmation

There is an important and useful setting in a KeePassXC entry under SSH tab, removing keys from agent when the database is locked (because of tight coupling with user login session, Gnome Keyring doesn't offer an option to the keys from the agent, which can theoretically increase the potential for their misuse. Original ssh-agent agent and specifically KeePassXC, both operating separately from the user login session, have options to automatically remove the keys from the agent). Enable it by checking:

  • Remove key from agent when database is closed/locked

It is worth noting, that KeePassXC additionally to storing the passphrase also offers to store the actual private key in the database as an attachment. With above setting enabled, further enabling user confirmation under the same SSH tab is a great hindrance when both, the key and its passphrase are stored inside the same database. To increase security, one should always strive to store different security factors on different locations (it is a safer approach to store the different factors of a security in different places. For instance, when using password + 2FA/MFA in a form of TOTP, store password in the manager and the other in the phone. This applies for SSH keys as well, for instance storing the private key in the filesystem /something I own/ and its passphrase it in a brain /something I know/. In practice, storing every available factor in single one secure .kdbx database with a strong master password is still marginally better than omitting some available factors, such as using just a password without TOTP or using a ssh key without a passphrase).

Furthermore, when user confirmation is enabled, but askpass is mis-configured, SSH will stop working, displaying an error:

sign_and_send_pubkey:
signing failed for RSA "[email protected]" from agent: agent refused operation

With the above in mind, I would advise considering enabling user confirmation only when either:

  • The actual private key and its passphrase are both stored in a different databases
  • Keys do not get automatically removed from agent when the database is locked

Preparation

Before making commands that require ssh automatically prompt for a KeePassXC database unlock dialog, there are some more steps required. Since there can only be one SSH agent active at a time, other agents possibly running have to be disabled or configured differently, before the KeePassXC agent implementation can be used.

chmod -x /usr/bin/gnome-keyring-daemon

General > Startup

  • Start only a single instance of KeePassXC
  • Minimize window after unlocking database
  • Remember previously used databases
    • Load previously open databases on startup

Security > Convenience

  • Lock databases when session is locked or lid is closed

SSH Agent

  • Enable SSH Agent integration
  • Configure an actual SSH entry accordingly:

Edit entry > Entry

Password: [KEY_PASSPHRASE]

Edit entry > SSH Agent

  • Add key to agent when database is opened/unlocked
  • Remove key from agent when database is closed/locked

Edit entry > SSH Agent > Private key

Insert either valid [Attachment] OR [External file]

Test the setup

Before continuing, test everything works properly. Start by unlocking the KeePassXC database and running ssh-add -l, the output should be:

4096 SHA256:9k/Nfk7fijei+JFj8F7YfyF7fhFHElSmpuFuew9+8f3 [email protected] (RSA)

Locking the database and running ssh-add -l should yield precisely this:

The agent has no identities.

If this is not the case, consider consulting the Further reading section at the bottom. There are useful links from the other people using KeePassXC to their advantage.

Prompt KeePassXC unlock with ssh

When everything is prepared and tested, the actual thing is to implement the script that gets called before any command employing ssh is run. To do so, create following two files:

  • Paste the following line into ~/.ssh/config:
ProxyCommand $HOME/.ssh/keepassxc-prompt %h %p
  • Last thing, create executable script ~/.ssh/keepassxc-prompt referenced above:
#!/bin/bash

until ssh-add -l &> /dev/null
do
  echo "Waiting for agent. Please unlock the database."
  keepassxc &> /dev/null
  sleep 1
done

/usr/bin/nc "$1" "$2"

Done! Running any command that relies on ssh while the database is still locked, the KeePassXC unlocking dialog is invoked. After unlocking, the keys are automatically added into the agent and the command succeeds. No more Permission denied (publickey). for the valid keys!

This is a 30th post of #100daystooffload.

Frequently Asked Questions (FAQ)

Why is the keepassxc-prompt script so plain?

It serves as a proof of concept. Modify as needed. One good modification is to add a timeout.

Isn't ProxyCommand used for SSH forwarding?

Yes, but it appears to be working with this approach without problems.

Why not just alias ssh to the wrapper script?

There are other commands that rely on ssh. With an alias every other command would not invoke the unlocking prompt, for instance git pull.

Why not replace /usr/bin/ssh with a wrapper script?

Many reasons. It prevents proper upgrades, with something security delicate as OpenSSH, it is advised to use updated software. Next, it requires on a single user's file, so it would disable ssh for other users on the system. Asides, it is plainly ugly.

Can I use D-Bus to detect if the database is locked/unlocked?

Sure. If there are also others keys in the agent that are managed separately of KeePassXC, simply use this instead of ssh-add -l in the keepassxc-proxy:

qdbus org.keepassxc.KeePassXC.MainWindow /org/freedesktop/secrets/collection/Passwords org.freedesktop.Secret.Collection.Locked

Could /etc/sshrc or, when enabled, ~/.ssh/rc be used for this purpose?

No. These hook files run on the server when the connection is initiated. For this to work, the hook must be run on the client. The latter also has to be enabled via PermitUserRC yes in /etc/ssh/sshd_config.

What are other situations where KeePassXC prompts for unlock automatically?

  1. When global Auto-type keyboard shortcut is used
  2. When KeePassXC Freesdesktop.org Secret Service is enabled and a program needs access
  3. On any browser plugin keyboard shortcut, assuming installed and configured properly

Settings > Browser Integration

  • Enable browser integration
    • Request to unlock database if it is locked