Stop giving a crap about keys and passwords

Seriously. Banish keys and passwords to where they belong.

I was going to post this on my (somewhat) defunct blog, and then decided…why? So here I am on Medium.

So: First post! What can I say, I moved twice (once across the entire country), bought a house, had a baby, lost my job, there was a pandemic…and….yeah.

OK, where were we.

I recently refreshed my Macs. This meant I went from a leased iMac Pro (via my old consulting company) to a new iMac and finally got around to getting a new laptop to replace my venerable clamshell.

I decided getting a new computer was a great time to rotate my SSH keys. I did so, set what I can only assume was a super clever password…and then promptly forgot the password days later.

I started wondering if there was a way to stop caring about keys. I had been meaning to move to ephemeral keys and SSH certificate authorities for some time, and it just never happened. My buddy Pete started using a Yubikey for OSX and told me how great it was, which started me down the path of coming up to speed on PIV and wondering if I could kill multiple birds with one stone.

I knew it was possible to use the Yubikey for SSH authentication, which itself was pretty neat — but I wondered if it was possible to combine OpenSSH user certificates with the process, so that it would never be necessary to update a remote host with new keys ever again.

My goal was pretty simple:

• Use the yubikey to login to the Mac, pin only.
• Use the yubikey to hold a new ECDSA key that will be used as the only SSH key. This should be a little more convenient, eg touch is sufficient.
• Sign this key with an SSH Certificate Authority, making it possible to have an easily usable backup yubikey for SSH, and also allowing for recovery from a truly dire situation where both Yubikeys are destroyed or inaccessible (eg: fire, my death or disablement, etc, etc)
• SSH should be two factor (eg requires pin at least once — possession of the Yubikey on its own is not enough).

I wrote a script to do all of the provisioning, since I ended up doing it multiple times tweaking stuff, and once I realized I really should have a failsafe yubikey, I decided it was worth the time to set it up.

First, we handle OSX login. You can either use the Yubikey Manager to do this with self signed certificates, or you can use the ykman command line tool. I used ykman because I wanted custom policy.

Thankfully, there’s a brew formula for it, and you’ll also get libykcs11.dylib along the way which can be used by most applications that support PKCS#11:

brew install ykman

For those not familiar with PIV, the standard defines a bunch of slots which can be used to store keys. There are four major ones: 9a, 9c, 9d and 9e. What happened to 9b? Who knows. You can read more about these slots on the Yubico dev site.

We start by generating two RSA keys and an ECDSA key. 9a/9d will be used for smartcard OS authentication. 9c will only be used for SSH (these code fragments are from the script).

# Generate two RSA certificates in 9a/9d for standard auth
echo “Generating 9a (authentication) key. You’ll be prompted for PIN.”
ykman piv keys generate --pin-policy default --touch-policy default 9a > ${NINEA}Pubkey.pem
echo “Generating 9d (key management) key. You’ll be prompted for PIN.”
ykman piv keys generate --pin-policy default --touch-policy default 9d > ${NINED}Pubkey.pem

We don’t care about the ECDSA x509 since we’re only using it for SSH — but some utilities really, REALLY expect a certificate to be available to fetch the public key, so it’s just easier to make a self signed one.

# Generate ECDSA key in slot 9c which will be used for ssh
echo “Generating 9c (digital signature / ssh) key. You’ll be prompted for PIN.”
ykman piv keys generate -a eccp384 --pin-policy once --touch-policy cached 9c > ${NINEC}Pubkey.pem
```

Note we use a different policy here. I didn’t want to go crazy, and I still want Ansible to (mostly) work and not make me pull my hair out. In practice, this means I authenticate with a PIN once, and touch the Yubikey to authorize sessions. That touch is cached long enough so that automation is generally not a problem and you aren’t constantly touching the key. OSX, on the other hand, requires a PIN generally every time you need to make an authentication (using 9a).

Now we generate CSRs for the 9a/9d slots, and sign the certificates:

# Generate CSRs for 9a/9d 
NINEA_SUBJ=”${CERT_SUBJECT},OU=${YUBISERIAL}-9a,CN=${1}”
NINEC_SUBJ=”${CERT_SUBJECT},OU=${YUBISERIAL}-9c,CN=${1}”
echo “Generating 9a CSR: ${NINEA_SUBJ}”
ykman piv certificates request \
 -s “${NINEA_SUBJ}” 9a ${NINEA}Pubkey.pem ${NINEA}Req.pem
echo “Generating 9c CSR: ${NINEC_SUBJ}”
ykman piv certificates request \
 -s “${NINEC_SUBJ}” 9d ${NINED}Pubkey.pem ${NINED}Req.pem
# Get certificates signed (alternatively, don’t be like me and use a
# self sign here)
citadel_sign ERISCOPIVCA ${NINEA} ${3} client
citadel_sign ERISCOPIVCA ${NINED} ${3} client
echo “Importing signed 9a certificate.”
ykman piv certificates import 9a ${NINEA}Cert.pem
echo “Importing signed 9d certificate.”
ykman piv certificates import 9d ${NINED}Cert.pem

You’ll note that the script refers to citadel, which is a USB Armory mk2 that I use for the purposes of an offline CA. It lives in a locked drawer, and before you make plans to burgle my house, the entire thing is encrypted, and, my house is alarmed.

Take that, 1337 haxxors.

You can feel free to modify the script to use self-signed certs instead. I have a PIV Certification Authority that is subordinate to a root CA; the PIV x509 reqs are signed by that.

The basic gist here is to simply get the certs signed, or self signed, and then make sure they’re trusted. For OSX, you must import the CA certificates into your keychain and fully trust them. This is especially important if you plan on making token authentication required — I haven’t, yet, and the only reason I haven’t is because every time I even come a little close to having to work with Apple’s MDM I run away screaming.

It’s a good idea to have more than one Yubikey in case one is lost / destroyed / microwaved / seized / vaporized / whatever. You can use the script to set up a failsafe key, which is what I did. Once the 9a/9d certs are provisioned, reinserting the Yubikey will result in your Mac asking if you want to pair (say yes, obviously). After that, you can use it for OSX login.

As a sidenote: I generally agree that working with the yubikey can be super confusing. Yubico has documentation somewhere between “fucking what” and “yeah, fine, ok” and has reinvented their management wheel (the tooling used to interact with the key) several times. The good news is that it seems to be trending better, but it still took me a while to get here.

OK, so now we have a bunch of keys. Cool.

Signing your Yubikey public key with an SSH user certificate authority

Now that we have the ECDSA key generated, we need a SSH user certificate authority. Rather than keep this key on a different Yubikey — which is absolutely a possibility — or the login Yubikey itself — it seemed like a better idea to generate and keep a copy of this offline (I use the aforementioned USB Armory for this purpose as mentioned above).

We can thank Theo for the fact that generating an SSH user CA is exactly the same as generating any random SSH key:

ssh-keygen -t ecdsa -b 521 -f user_ssh_ca

(You don’t have to use ECDSA here, you can use any algorithm you like to sign your pubkey)

Lastly, we can sign our public key with our shiny new user CA:

ssh-keygen -s user_ssh_ca -I baughj -n baughj,justin.baugh -V +52w -z ${SSH_CERT_SERIAL} reqs/${NINEC}.pub

-n specifies the principals (usernames) allowed to authenticate. Note that if you intend on using this for root auth, you should put root in the list of principals. I fully intended to use this opportunity to rid myself finally and forever of the awful bad habit of using root — so I didn’t. I use OATH instead, for sudo, with the aforementioned yubikey (see Getting Rid of Root below).

Note that I set the validity to 52 weeks here. No reason to not rotate the keys, especially when it is so easy to do so.

OK, so now we have the SSH certificate. We’re done, right?

Configuring & Using yubikey-agent

Not yet. Now you need to install yubikey-agent, which is also helpfully in brew:

brew install yubikey-agent

If you want to do what I did regarding having a different policy for 9a/9d/9c, you can also use my fork which changes the agent to use 9c (this is sadly not currently configurable) and also makes the notification to touch a little bit faster.

Once you’ve got yubikey-agent installed, you can use `brew services` to install a launchd plist to make sure it is always running. Since I am running a fork, I have a pretty simple plist I use to run it out of my homedir:

Once it’s running, you’ll also need to modify your local SSH config (eg ~/.ssh/config) to use it:

# Use yubikey-agent
IdentityAgent /usr/local/var/run/yubikey-agent.sock

You will also need to copy your signed SSH pubkey to id_ecdsa-cert.pub in your .ssh directory, and I also keep the pubkey at id_ecdsa.pub as well. This means you can use either (the ECDSA key or the signed ECDSA key). I fall back to using the ECDSA pubkey for stuff that doesn’t support cert authorities (like Github, which weirdly supports it in Github Enterprise Hyper On-Prem Cloud Cool Ranch Edition, but not for regular ol’ Github).

Once you’ve done this, if you crank up verbosity on SSH, you’ll see it uses the Yubikey-stored key, and the certificate. You may get prompted for a PIN and/or you’ll have to touch the Yubikey.

debug1: Offering public key: /Users/baughj/.ssh/id_ecdsa ECDSA-CERT SHA256:Mi7uL/1JiE7idBWcaNRLxS1rkFrumA4trhPrgF05yLQ
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /Users/baughj/.ssh/id_ecdsa ECDSA-CERT SHA256:Mi7uL/1JiE7idBWcaNRLxS1rkFrumA4trhPrgF05yLQ

Handling removal of the token

If you feel like dealing with MDM, you can attach an event to the token removal. You can also configure OSX to lock automatically on token removal (via System Preferences -> Security & Privacy -> Advanced) but in my experience so far, this works intermittently.

Since I don’t feel like climbing the MDM hill just yet, I modified this original script by Nick Irvine to detect when the screen is locked as well as when the yubikey is removed. It is a really dumb hacky thing that gets the job done:

When the yubikey is removed the screensaver is automatically started (you DO have OSX set to require reauth on screensaver starting, right?), and when the screen is locked but the key is still present (say I walk away momentarily), we send a HUP to yubikey-agent to release the key (see Downsides below).

Getting rid of root

Part of my motivation for this as well was to finally kill my longstanding (awful) habit of using root fairly liberally. This is just ingrained from being a sysadmin for 20+ years and a DeVoPs PrOfEsSiOnAl for the last eight or so.

Depending on your operating system, this is pretty easy. For FreeBSD you’ll need oath-toolkit. Debian/Ubuntu et al need pam_oath.

I wanted to start by disabling the native OTP app on the Yubikey, which seems to only support SHA1. I wanted to use the OATH application instead. I ended up going down a Yubico rathole here: In order to bind it to a touch, you’ll have to get the Yubikey personalization tool, but wait, here is the beginning of the hole: If you install yubikey-personalization from brew like you might be inclined, and try to use it, you’ll get the following unhelpful error:

Yubikey core error: no yubikey present

(except ykman shows it, so…)

It turns out that the personalization CLI tool is, well, abandoned, and you should use the GUI instead. OK, sure.

…Then it turns out the personalization tool doesn’t support the 5 series. At this point, my WAT is growing.

After some reading, it turns out that the Yubikey doesn’t support TOTP natively from press because it has no internal clock, and I don’t really want to use HOTP because I don’t trust myself to never accidentally touch the key. So, we’ll have to live with generating codes at the command line, like goddamn animals.

To generate a TOTP key:

head -10 /dev/urandom | sha256sum | cut -b 1–30 | oathtool -v -d 8 — totp -

(The — tells oathtool to read the key from stdin. We don’t want it in our history.)

Now with that key, we can update our users.oath file (either /etc/users.oath on Linuxes or /usr/local/etc/users.oath on FreeBSD:

HOTP/T30/8 baughj — <my shiny oath key>

Your oath key here should be the value returned from oathtool (the Hex secret).

Once we’ve done this, we can program the Yubikey:

[baughj@starfire ~]$ ykman oath accounts add -o totp -d 8 -a sha1 -i erisco -t baughj-sudo
Enter a secret key (base32): *”Base 32 secret” value from oathtool*

Note that -t makes it so the Yubikey must be touched to generate a code. You can leave this off if desired so only presence of the Yubikey is needed.

Now, we can use ykman to retrieve TOTP codes:

[baughj@starfire ~]$ ykman oath accounts code baughj-sudo
Touch your YubiKey…
erisco:baughj-sudo 40966078

Lastly, we need to add the following line to our PAM configuration. On FreeBSD, at least, I only allow OATH for sudo, so the change is literally as simple as creating /etc/pam.d/users.oath and adding the line below. YMMV for other configurations. We specify the path to the users.oauth file as above, and set the window to 30 seconds for code validity:

auth required /usr/local/lib/security/pam_oath.so usersfile=/usr/local/etc/users.oath window=30

Hooray! No more root! And I can conveniently ignore the fact that it took me so long to do this in the first place.

The downsides

Although this solution works really well and I am now using it on both of my Macs, there are some warts. yubikey-agent is great stuff…except: it relies on piv-go, which is not currently very good at sharing. Effectively, the moment you enter your PIN, piv-go opens an exclusive session with the Yubikey, and won’t release it.

(Realtime view of piv-go in operation)

This means you have to do some absolutely insane workarounds:

• With OSX logins, if you lock the screen but don’t remove the Yubikey — for instance, you get up to go do something quickly — you’ll find that yubikey-agent won’t let OSX use the key to reauthenticate. Womp. This has the awesome side effect of locking the entire machine up for a bit until you remove the key or OSX’s attempt to access the key times out.
• Similarly with local OSX terminal usage, sudo will need access to the Yubikey. This leads to the mind-bendingly awful workaround herein: alias sudo=’killall -HUP yubikey-agent; sudo’

Yeah. I hate it too. With fire. I even hated typing it here. I hated publishing it even more.

The solution for one of these problems (locking the screen but not yanking the Yubikey) here is to use the aforementioned osx-lock-on-remove-yubikey.sh via a launchd plist. If the script detects that the key is removed OR the screen is locked, it sends a HUP to the agent, which makes it release its exclusive lock. You’ll still have to occasionally do this yourself (eg to locally sudo, or change a setting / install software) which is extremely annoying, but I’m willing to put up with it until the piv-go developers come up with a permanent fix.

Troubleshooting

Sudo (on remote machines) doesn’t work

You can turn on debug by adding `debug` to the PAM line. This will give you a bunch of debug information when you attempt to use sudo. You’ll also need to make sure that your remote systems have correct time, which is something you should be doing anyway, but being out of sync can cause failures.

OSX won’t stop asking me for keychain passwords, even after PIN login

If you keep getting prompted by OSX for keychain passwords after a PIN login with a Yubikey, try running sudo security authorizationdb smartcard enable. This fixed it for me.

Conclusion

Yubikey in the wild, connected to phoenix. Having it on a magnetic quick release has been super useful.

I hope this was useful. If not, I’d welcome suggestions. This has allowed me to increase my local security while at the same time reducing the number of things I need to remember or care about. It’s rare those two things occur together.