Fix Consul reboot loop

Fix Consul reboot loop

I've recently encountered an issue with HasiCorp's Consul being stuck in a boot loop.

Roman Zipp, October 6th, 2023

I've recently encountered an issue with HasiCorp's Consul being stuck in a boot loop.

The Issue

Although the following lines may not be directly correlated with the issue, I chose to include them just in case anyone - like me - searches for those errors and actually has the same - but different problem.

Systemd reports

systemd[1]: consul.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: consul.service: Failed with result 'timeout'.
systemd[1]: Failed to start "HashiCorp Consul - A service mesh solution".

Application logs

consul[1669055]: 2023-10-05T17:50:23.579Z [ERROR] agent: error handling service update: error="error watching service config: No cluster leader"
consul[1669055]: 2023-10-05T17:50:24.979Z [ERROR] agent.proxycfg: Failed to handle update from watch: kind=connect-proxy proxy=_nomad-task-...>
consul[1669055]: 2023-10-05T17:50:25.531Z [WARN]  agent.leaf-certs: handling error in Manager.Notify: error="CA is uninitialized and unable to sign certificates yet: provider is nil" index=1

Solution

Add the current server IP address to the retry_join array in your consul.hcl config.

retry_join = ["10.0.0.1"]

Here you go.