These past few weeks have been quite hectic and interesting.  I am working on a very large project to roll Chef out to a major client.  Part of the goal is to improve compliance in the environment and by using Chef, we will standardize on the delivery of compliant servers based on the client’s requirements.  Recently, we have started integrating Hashicorp Vault into the solution and I plan to cover some of my experiences intermingling Chef with Hashicorp Vault.  A few days ago, I made a mistake in configuring ChefVault that I want to share.

For those unfamiliar with ChefVault, it is a way to create encrypted Data Bags (JSON type objects containing details usable in Chef recipes).  These ChefVault items use a method of encryption where the keys to unlock are based on client or user PEM files.  When configured, you can allow specific hosts to gain access to the encrypted items while not requiring the ability to send the decryption key (the SALT) to the host.  Instead, the decryption will use the client PEM file to authenticate.  This is awesome as I can constantly rotate-keys and change access without the need to visit every machine to update a file on the host.

As I played with the configuration, I accidentally locked the admin accounts out of the ChefVault item.  The ChefVault configuration allows for two types of access – admin/FullControl and client/ReadOnly.  In testing things, I mistakenly removed both of the admin accounts (fast typing and not paying attention to what I was doing) and in doing so, could no longer edit the item.  Since I couldn’t edit the files, I needed a quick and easy way to recover from my mistake.

I had accidentally remove all admins and if I tried to view the Chef vault item, it would fail with the following error:

$> knife vault showexosphere build_team -M client
ERROR: ChefVault::Exceptions::SecretDecryption:exosphere/build_team is not encrypted 
with your public key. Contact an administrator of the vault item to encrypt for you!

How do I delete chef-vault items without the admin keys

Note: Following these steps will delete the chef-vault item permanently and requires that the item be recreated.  Only do this if you are completely locked out of editing the file. 

The first step is to log into a system that can still see the keys.  Fortunately, I had added my Chef account as a client so I am able to see the data.

$> knife vault show exosphere build_team -M client -Fjson -pall

{
 "id": "build_team",
 "address": "<some-secret-fqdn>",
 "token": "<an-api-token>",
 "search_query": "name:**", 
 "admins": [
 ],
 "clients": [
   "chefautomate.exospheredata.lab",
   "buildrunner01.exospheredata.lab",
   "buildrunner02.exospheredata.lab",
   "rhel-68-base-acceptance.exospheredata.lab",
   "rhel-73-base-acceptance.exospheredata.lab",
   "windows-2012r2-base-acceptance.exospheredata.lab",
   "windows-2016-base-acceptance.exospheredata.lab"
 ]
}

By using the -Fjson format, I can copy this information for later use.  I will need to create the item and getting back everything in that format is helpful. The field -pall will return the other important details:

• Which admins are currently assigned to control the item
• Which hosts have client access
• What search terms are used if any

Note: If your Vault item has ANY admin listed, then contact them to re-add you.  The following steps should only performed if you are permanently locked out of the chef-vault item.

Once we have backed up the ChefVault item, we can easily remove the item and recreate everything.  The next few steps will show how to forcibly remove the chef-vault item and then clean it up.  ChefVault leverages the native Chef data_bag construct but in a very special way.  Instead of having just a single, clear-text data_bag,  ChefVault will create an encrypted bag_item and a special bag item with the suffix _keys.

# You can verify that the details of the item by running
$> knife data_bag show exosphere
build_team
build_team_keys

# First we will remove the encrypted item completely
$> knife data_bag delete exosphere build_team -y

# Next we will remove the special _keys item
$> knife data_bag delete exosphere build_team_keys -y

# Ensure that everything is cleared, we can try to show the ChefVault item again
$> knife vault show exosphere build_team -M client -Fjson -pall
ERROR: ChefVault::Exceptions::KeysNotFound: exosphere/build_team_keys could not be found

# Before recreating the key, I saved the details of the bag to a file called secrets.json
{
 "id": "build_team",
 "address": "<some-secret-fqdn>",
 "token": "<an-api-token>"
}

# Now we recreate the ChefVault item using the details that we backed up.
$> knife vault create exosphere build_team -S "name:**" -A "delivery,admin" -J secrets.json -M client

# Note: I am using the search term 'name:**' which allows all nodes to become clients so I will
# not need to re-add each client.  Otherwise, I could provide a comma-delimited list using 
# the -C option.

Hopefully, this article will help you if you too mistakenly lock out a ChefVault bag.