Ever misplace the key to your house?  It really is annoying when you need to get back into your house. Even more so when you realize that you left your 8 week old puppy loose. If you call the locksmith, he will likely re-key your lock so you can get into the house but what about when your wife, kids, mother, uncle, etc also have a copy of your key.  You probably called them all but unable to reach them, you went with the Hammer of Thor approach and called the locksmith.  What happens when you ultimately find your keys and want to change the locks back?

Recently, we had an issue were an admin lost a copy of PEM file for the service account on a Chef server.  This admin decided to reset the key instead of creating a new account or even reaching out to the other holders of the key.  It became an annoyance for all of us because we encrypted the Chef-Vault items using that service account and our automation to push the cookbooks to the organizations on that single Chef Server also uses that account.  Fortunately, we were able to recover from this snafu.

Here are the steps to reset the public key of a Chef user (or client key for that matter) back to match your existing private key.  These steps assume that your account can SSH into the Chef Server and has server admin rights:

# Regenerate your public key using the local private PEM using openssl. This command 
# assumes that you are in the root of your chef-repo and that the key is in the 
# default .chef directory

$> openssl rsa -in .chef/service_account.pem -pubout > .chef/service_account.pub

# Once the RSA public key is generated, we need to secure copy the file onto the Chef Server

$> scp ./chef/service_account.pub [email protected]:~/service_account.pub

# Next, you will need to SSH into the Chef Server

$> ssh [email protected]

# After logging in, you can execute the following commands to replace the current 
# private key with your new key.

# First we will remove the new public key by deleting the default key.  Then we 
# will recreate the default key with the new public file

chefserver> chef-server-ctl user-key-delete service_account default

chefserver> chef-server-ctl user-key-add service_account admin -key-name default --public-key ~/service_account.pub

# Now, we should receive no errors and be able to test using that key again on our 
# local machine

$> knife node list -u service_account -k .chef/service_acount.pem

Hopefully these steps will help you if you lose your key.  As part of our plans to remediate this, we have worked on a new Chef cookbook that will build our servers and keep them in the right state.  We have integrated the service_account key creation portion into the recipe and are pulling the key directly from Hashicorp Vault.  We are even leveraging our new secrets_management cookbook (recently published to the Chef Supermarket) to handle the secrets.