Storage encryption with Exoscale Flexible Storage template, LUKS and HashiCorp Vault

Exoscale Flexible Storage template empowers users to resize and/or create disk partitions as they deem fit, thanks to the flexibility provided by the Linux Logical Volume Manager (LVM).

In this article, we will leverage this flexibility to create a new partition, encrypted using Linux Unified Key Setup (LUKS), which passphrase shall be secured using HashiCorp Vault.

To achieve this objective, we will, in the following sections:

• configure HashiCorp Vault to:
• provide Encryption-as-a-Service to secure the LUKS passphrase

• perform seamless authentication of compute instances, using Exoscale Vault Authentication Plugin

• create and encrypt a new LVM Logical Volume (LV) / partition using LUKS

• setup the bits’-n-pieces (script, systemd unit) required to automatically - yet securely - enable the LUKS encrypted partition at boot time and disable it on shutdown

All of it using Exoscale CLI to interact with Exoscale cloud platform.

Configure your Vault server

We will assume that you already have a running Vault server available (which setup is out of the scope of this article).

Vault Encryption-as-a-Service

In order to secure the LUKS passphrase, we will use Vault Transit Secrets Engine, enabling so-called Encryption-as-a-Service (EaaS):

# Enable Vault Transit Secrets Engine
$ vault secrets enable transit
# [output]
Success! Enabled the transit secrets engine at: transit/

Along a LUKS (passphrase) dedicated encryption/decryption endpoint/key and ad-hoc Policy:

# Create LUKS-dedicated encryption/decryption endpoint/key
$ vault write -f transit/keys/luks
# [output]
Success! Data written to: transit/keys/luks

# Create ad-hoc Policy
$ vault policy write 'transit-luks' - <<EOF
# Allow Policy holder to encrypt LUKS passphrase
path "transit/encrypt/luks"
{
    capabilities = ["update"]
}

# Allow Policy holder to decrypt LUKS passphrase
path "transit/decrypt/luks"
{
    capabilities = ["update"]
}
EOF
# [output]
Success! Uploaded policy: transit-luks

Vault Exoscale Authentication Plugin

Vault Exoscale Authentication Plugin allows to authenticate Exoscale Compute Instances based on properties of each instance, whether intrinsic (set by the Exoscale cloud platform; e.g. the instance UUID) or assigned by the users (such as instance tags).

Dedicated Exoscale IAM credentials

In order to access Exoscale API and verify/match compute instances properties, the Vault Exoscale Authentication Plugin must be configured with Exoscale IAM credentials, ideally scoped specifically for the purpose at hand:

# Create Vault-specific Exoscale IAM credentials
$ exo iam apikey create vault-plugin-auth-exoscale \
  --operation 'compute/listZones,compute/listVirtualMachines'
# [output (partial)]
Name    vault-plugin-auth-exoscale
Key     EXO...
Secret  ...

Mark the output Key and Secret as we will need them in the next section.

Setup the Vault Exoscale Authentication Plugin

As any other Vault plugin, the Vault Exoscale Authentication Plugin must be registered before it can be used. Make sure to add the plugin_directory = "/usr/lib/vault/server/plugins" to your Vault server’s configuration. Then:

# Register the Exoscale Authentication Plugin
$ vault plugin register \
  -command="vault-plugin-auth-exoscale" \
  -sha256="$(sha256sum /usr/lib/vault/server/plugins/vault-plugin-auth-exoscale | cut -d " " -f 1)" \
  auth exoscale
# [output]
Success! Registered plugin: exoscale

You may then enable and configure it with the Exoscale IAM credentials we created above:

# Enable the Exoscale Authentication Plugin
$ vault auth enable exoscale
# [output]
Success! Enabled exoscale auth method at: exoscale/

# Configure the Exoscale Authentication Plugin
$ vault write auth/exoscale/config api_key=EXO... api_secret=...
$ vault read auth/exoscale/config
# [output]
Key             Value
---             -----
api_endpoint    https://api.exoscale.com/v1
api_key         EXO...
api_secret      ...

Configure a Vault Exoscale Authentication Role

In order to grant our storage instance the proper permissions, we shall: - authenticate it using the validator parameter and a CEL expression that matches any property you may deem fit; e.g. the instance name - authorize it by granting it the transit-luks Policy (token_policies)

$ vault write auth/exoscale/role/storage \
  validator='instance_name.startsWith("storage")' \
  token_period='24h' \
  token_policies='transit-luks'
# [output]
Success! Data written to: auth/exoscale/role/storage

Mark the Token being a Periodic Token, which will come handy for the automation part (see section further below).

Create and encrypt a new LVM Logical Volume / partition

Please refer to the Exoscale Flexible Storage template documentation to create an Exoscale Flexible Storage instance and get acquainted with its (re)partitioning.

Authenticate with the Vault server

Let’s start by authenticating with the Vault server:

# Retrieve a Vault Token ("login")
# (NB: at the time of writing 'vault login -method=exoscale' is not yet supported)
$ umask 077
$ vault write -field=token auth/exoscale/login \
  role=storage \
  instance=$(wget -qO- http://metadata.exoscale.com/1.0/meta-data/instance-id) \
  zone=$(wget -qO- http://metadata.exoscale.com/1.0/meta-data/availability-zone) \
> ~/.vault-token

# Verify login/permissions
$ vault token lookup
# [output (partial)]
Key                 Value
---                 -----
display_name        exoscale
path                auth/exoscale/login
policies            [default transit-luks]

Create the LUKS passphrase

We may now create a random LUKS passphrase and secure it with Vault:

# Create and Vault-encrypt the LUKS passphrase
$ openssl rand -base64 24 \
| vault write -field=ciphertext transit/encrypt/luks plaintext=- \
| tee /etc/luks.passphrase
# [output]
vault:v1:lX3HqaqNdKbB2uD4yfcK8CfWsv/ugRFrTzJSrRFiuowpluFJ9nyGKbPq+HiyJINaq0SOEw==

# Decrypt and backup (!) the LUKS passphrase
$ cat /etc/luks.passphrase \
| vault write -field=plaintext transit/decrypt/luks ciphertext=-
# [output]
kDTAmQFldH0J0yhu3M2zfvHTOZLJMtTY  # !!! SECRET !!!

Mark the /etc/luks.passphrase being encrypted by Vault and the actual LUKS passphrase being decrypted on demand by Vault (provided a successful login and proper permissions).

Create a new LVM Logical Volume for LUKS

Creating the LUKS-dedicated LVM Logical Volume (LV) / partition is straight-forward:

$ lvcreate -n lv.luks -L 10G vg.flex
# [output]
  Logical volume "lv.luks" created.

Encrypt the LVM Logical Volume with LUKS

We may now encrypt the new Logical Volume using LUKS:

# Install LUKS utilities
$ apt-get install cryptsetup-bin

# Fill the new partition with random data
# (optional but recommended; takes time!)
$ dd if=/dev/urandom of=/dev/mapper/vg.flex-lv.luks bs=4k

# Initialize the LUKS partition
$ cat /etc/luks.passphrase \
| vault write -field=plaintext transit/decrypt/luks ciphertext=- \
| cryptsetup luksFormat --batch-mode --type luks2 --pbkdf argon2id --key-file - /dev/mapper/vg.flex-lv.luks

# Show the LUKS partition status
$ cryptsetup luksDump /dev/mapper/vg.flex-lv.luks
# [output (partial)]
LUKS header information
Version:        2
Epoch:          3
Metadata area:  16384 [bytes]
Keyslots area:  16744448 [bytes]

And “open” it:

# Open the LUKS partition
cat /etc/luks.passphrase \
| vault write -field=plaintext transit/decrypt/luks ciphertext=- \
| cryptsetup open --type luks --key-file - /dev/mapper/vg.flex-lv.luks luks

We can now format its filesystem (example given with ext4):

$ mkfs.ext4 -L LUKS /dev/mapper/luks
# [output (partial)]
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 2617344 4k blocks and 655360 inodes

And mount it:

# Create the encrypted data mountpoint
mkdir -p /luks

# Mount the encrypted partition
mount /dev/mapper/luks /luks

# Show the partition capacity/usage
df -h /luks
# [output]
Filesystem        Size  Used Avail Use% Mounted on
/dev/mapper/luks  9.8G   37M  9.3G   1% /luks

Mark /dev/mapper/luks being the (encrypted) partition to use (not /dev/mapper/vg.flex-lv.luks).

Boot and shutdown automation

In order to automate the various steps to open, mount, unmount and close the LUKS partition, we will need a control script and systemd unit, which shall open and mount the LUKS partition on start, as well as unmount and close it on stop.

Ideally, we would also use the Vault Agent to automatically authenticate our compute instance using the Vault Exoscale Authentication Plugin and act as proxy for the required vault commands.

For the sake of simplicity, we’ll stick to “manually” authenticating with the Vault server, within the control script itself.

Control script and systemd unit

Let’s start with a small shell script to perform start, stop and status actions; in /usr/local/bin/luks-partition:

#!/bin/sh
set -e

case "${1}" in

  'start')
    # Log into Vault
    if ! test -e ~/.vault-token && [ -z "${VAULT_TOKEN}" ] && [ -z "${VAULT_AGENT_ADDR}" ]; then
      VAULT_TOKEN="$(
        vault write -field=token auth/exoscale/login \
              role=storage \
              instance=$(wget -qO- http://169.254.169.254/1.0/meta-data/instance-id) \
              zone=$(wget -qO- http://169.254.169.254/1.0/meta-data/availability-zone)
      )"
      export VAULT_TOKEN
    fi

    # Open the LUKS partition
    if ! test -e /dev/mapper/luks; then
      cat /etc/luks.passphrase \
      | vault write -field=plaintext transit/decrypt/luks ciphertext=- \
      | cryptsetup open --type luks --key-file - /dev/mapper/vg.flex-lv.luks luks
    fi

    # Mount the LUKS partition
    if ! mountpoint -q /luks; then
      mkdir -p /luks
      mount /dev/mapper/luks /luks
    fi
    ;;

  'stop')
    # Unmount the LUKS partition
    if mountpoint -q /luks; then
      umount /luks
    fi

    # Close the LUKS partition
    if test -e /dev/mapper/luks; then
      cryptsetup close luks
    fi
    ;;

  'status')
    mountpoint /luks
    cryptsetup status luks
    ;;

esac

And the corresponding systemd unit; in /etc/systemd/system/luks-partition.service:

[Unit]
Description=Start/stop the LUKS-encrypted partition
After=local-fs.target vault.service

[Service]
Type=oneshot
RemainAfterExit=yes
Environment=VAULT_ADDR=http://127.0.0.1:8200
ExecStart=/usr/local/bin/luks-partition start
ExecStop=/usr/local/bin/luks-partition stop

[Install]
WantedBy=multi-user.target

Which we can now enable and start:

# Reload systemd configuration
$ systemctl daemon-reload

# Enable the LUKS partition control unit
$ systemctl enable luks-partition

# Start the LUKS partition control unit
$ systemctl start luks-partition

# Query the LUKS partition control unit status
$ systemctl status luks-partition
# [output]
● luks-partition.service - Start/stop the LUKS-encrypted partition
   Loaded: loaded (/etc/systemd/system/luks-partition.service; enabled; vendor preset: enabled)
   Active: active (exited) since Fri 2021-07-02 07:07:28 UTC; 3s ago
  Process: 21515 ExecStart=/usr/local/bin/luks-partition start (code=exited, status=0/SUCCESS)
 Main PID: 21515 (code=exited, status=0/SUCCESS)

Should you have a service - e.g. MariaDB database - that has its data on the LUKS partition, just make sure to set the proper After=luks-partition.service dependency in its systemd unit or as an override. Example given in /etc/systemd/system/mariadb.service.d/after-luks-partition.conf:

[Unit]
After=luks-partition.service