Gossip Encryption
This topics describes how to enable gossip encryption on a Consul datacenter.
Enable gossip encryption
We recommend enabling gossip encryption to all new deployed Consul datacenters.
If you have an existing datacenter running Consul 0.8.4
and above, it is possible to modify its configuration to support gossip encryption.
Below are listed the steps required for both scenarios:
- Use
consul keygen
to generate a new gossip encryption key. - Create a configuration file that includes the
encrypt
parameter set to the newly generated key. - Distribute the configuration file to all the agent nodes that need to be pert of the datacenter and start the Consul agent on all the nodes.
- Use
consul keygen
to generate a new gossip encryption key. - Create a configuration file that includes the
encrypt
parameter set to the newly generated key andencrypt_verify_incoming
andencrypt_verify_outgoing
set tofalse
. - Distribute the configuration file to all the agent nodes that need to be pert of the datacenter and perform a rolling restart of all the agents.
- Update the
encrypt_verify_outgoing
setting totrue
and perform a rolling restart of all the agents. - Update the
encrypt_verify_incoming
setting totrue
and perform a rolling restart of all the agents.
If you have multiple datacenters joined in WAN federation, be sure to use the same encryption key in all datacenters.
Enable gossip encryption on a new datacenter
Enable gossip encryption on a new datacenter is a straightforward process and should be the default approach for all new datacenters you are deploying. To enable gossip encryption you set an encryption key when starting the Consul agent. The key can be set via the encrypt
parameter.
Step 1: Generate an encryption key using consul keygen
.
$ consul keygen
YwgWlBvicJN17UOYcutXLpJSZsw5aWpLEEWqgK635zA=
You can generate a new gossip key using any method that can creates 32 random bytes encoded in base64.
For example on Linux you can use openssl
or dd
to create one.
Step 2: Create a configuration file that includes the encrypt
parameter set to the newly generated key.
encrypt = "YwgWlBvicJN17UOYcutXLpJSZsw5aWpLEEWqgK635zA="
Step 3: Distribute the configuration file to all the agent nodes that need to be pert of the datacenter and start the Consul agent on all the nodes.
If gossip encryption is properly configured Gossip Encryption: true
will be shown in the logs at startup.
consul.log
==> Starting Consul agent...
Version: '1.19.0'
Build Date: '2024-06-12 13:59:10 +0000 UTC'
Node ID: 'e74b1ade-e932-1707-cdf1-6579b8b2536c'
Node name: 'consul-server-0'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: false)
Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: 8443, gRPC: -1, gRPC-TLS: 8503, DNS: 53)
Cluster Addr: 172.19.0.7 (LAN: 8301, WAN: 8302)
Gossip Encryption: true
Auto-Encrypt-TLS: true
ACL Enabled: true
Reporting Enabled: false
ACL Default Policy: deny
HTTPS TLS: Verify Incoming: false, Verify Outgoing: true, Min Version: TLSv1_2
gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
Internal RPC TLS: Verify Incoming: true, Verify Outgoing: true (Verify Hostname: true), Min Version: TLSv1_2
## ...
Enable gossip encryption on an existing datacenter
Gossip encryption can also be enabled on existing datacenters, but requires several extra steps.
Step 1: Generate an encryption key using consul keygen
.
$ consul keygen
YwgWlBvicJN17UOYcutXLpJSZsw5aWpLEEWqgK635zA=
Step 2: Create a configuration file that includes the encrypt
parameter set to the newly generated key. Set encrypt_verify_incoming
and encrypt_verify_outgoing
to false
.
/etc/consul.d/encryption.hcl
encrypt = "YwgWlBvicJN17UOYcutXLpJSZsw5aWpLEEWqgK635zA="
encrypt_verify_incoming = false
encrypt_verify_outgoing = false
Step 3: Distribute the configuration file to all the agent nodes that need to be pert of the datacenter and initiate a rolling update of all the
agents with these new values. After this step, the agents will be able to decrypt gossip but will not yet be able to send encrypted traffic. A rolling update can be made by restarting the Consul agents (clients and servers) in turn. consul reload
or kill -HUP <process_id>
is not sufficient to change the gossip configuration.
Step 4: Update the encrypt_verify_outgoing
setting to true
and perform another rolling update of all the agents by restarting Consul on each agent. The agents will now be sending encrypted gossip but will still allow incoming unencrypted traffic. Complete the process on all the nodes before moving to the next step.
/etc/consul.d/encryption.hcl
encrypt = "YwgWlBvicJN17UOYcutXLpJSZsw5aWpLEEWqgK635zA="
encrypt_verify_incoming = false
encrypt_verify_outgoing = true
Step 5: Update the encrypt_verify_incoming
setting to true
and perform a final rolling update on all the agents.
/etc/consul.d/encryption.hcl
encrypt = "YwgWlBvicJN17UOYcutXLpJSZsw5aWpLEEWqgK635zA="
encrypt_verify_incoming = true
encrypt_verify_outgoing = true
Rotate the gossip encryption key
It is important to periodically rotate the gossip encryption key used by your Consul datacenter.
The process of rotating the gossip encryption key is centralized and can be performed on a single datacenter node.
The steps to rotate a gossip encryption key are listed below:
- Generate a new encryption key using the
consul keygen
command. - Install the new encryption key using the
consul keyring -install
command. - Instruct Consul to use the new key with the
consul keyring -install
command. - Verify the new key is installed in your Consul datacenter with the
consul keyring -list
command. - Remove the old key using the
consul keyring -remove
command.
Generate a new encryption key
Generate a new key using consul keygen
:
$ consul keygen
FfRV9j6NXU9LlCI4zLZjjpZdj4Nrqsdm7R8YgzSHzHw=
Add new key to the keyring
Add your newly generated key to the keyring.
$ consul keyring -install FfRV9j6NXU9LlCI4zLZjjpZdj4Nrqsdm7R8YgzSHzHw=
==> Installing new gossip encryption key...
Verify that the new key is installed
Once you have added the key to one of the Consul agents, it will be propagated across the whole datacenter. You do not need to repeat the command on other agents.
You can ensure that the key has been propagated to all agents by verifying the number of agents that recognize the key over the number of total agents in the datacenter.
$ consul keyring -list
==> Gathering installed encryption keys...
WAN:
FfRV9j6NXU9LlCI4zLZjjpZdj4Nrqsdm7R8YgzSHzHw= [1/1]
YwgWlBvicJN17UOYcutXLpJSZsw5aWpLEEWqgK635zA= [1/1]
dc1 (LAN):
YwgWlBvicJN17UOYcutXLpJSZsw5aWpLEEWqgK635zA= [7/7]
FfRV9j6NXU9LlCI4zLZjjpZdj4Nrqsdm7R8YgzSHzHw= [7/7]
You must check that the two keys are installed in the datacenter, and are recognized by all agents, as well as by all the server agents. The server agents are listed in the WAN
section. Do not proceed to the next step unless all agents have the new key.
Promote the new key to primary
Once all agents have received the key and are able to use it as the primary encryption key, it is possible to promote the new key to primary.
$ consul keyring -use FfRV9j6NXU9LlCI4zLZjjpZdj4Nrqsdm7R8YgzSHzHw=
==> Changing primary gossip encryption key...
Remove the old key from the keyring
To avoid unused keys remaining in the keyring, we recommended you remove the old primary from the keyring once a new key is installed.
$ consul keyring -remove YwgWlBvicJN17UOYcutXLpJSZsw5aWpLEEWqgK635zA=
==> Removing gossip encryption key...
Verify that the keyring contains only one key.
$ consul keyring -list
==> Gathering installed encryption keys...
WAN:
FfRV9j6NXU9LlCI4zLZjjpZdj4Nrqsdm7R8YgzSHzHw= [1/1]
dc1 (LAN):
FfRV9j6NXU9LlCI4zLZjjpZdj4Nrqsdm7R8YgzSHzHw= [7/7]
Next steps
Documentation for the commands used in this topic is available at Consul agent configuration - Encryption Parameters. You can find more information over the gossip protocol used by Consul at Gossip Protocol.
After enabling gossip encryption, to continue securing your Consul datacenter, enable mutual TLS encryption. Read more on Mutual TLS (mTLS) Encryption.
To learn how to automate gossip key rotation using HashiCorp Vault and consul-template, refer to the Automatically Rotate Gossip Encryption Keys Secured in Vault tutorial.