managing Let’s Encrypt wildcard certificates

For a long time, TLS (née SSL) certificates were out of reach for the low-budget network admin. They got cheaper and easier to get, but it was still a hassle and an expense. Then came Let’s Encrypt, which offered no-cost certificates that you could obtain using fully automated tools.

I moved (almost) all my sites to TLS, and life was good but for one thing: they didn’t offer wildcard certificates, which meant I needed a cert for every subdomain. Dozens of certs. And they expired every three months. It meant that every few days I was having to renew certs on different machines. Due to some technical issues, it was difficult for me to automate the process.

Early this year (2018), Let’s Encrypt started offering wildcard certificates, so I would only need one cert for every service on a given domain. I was an early adopter, and went from needing dozens of certs to just nine.

I thought all was good, but then I realized that those nine certs had to be distributed to dozens of servers and, again, it had to be done every 90 days. This time, though, it was fairly easy to automate the process.

In case there’s anyone out there in a similar situation, here are some sample scripts.

Renewing the Certs

One of my servers is set up to be the home for the certs. It runs Linux (Devuan or Ubuntu server) and has certbot-auto installed from Let’s Encrypt. This script will attempt to renew each expiring cert. Wildcard certs can only be authenticated using DNS, and I use a multiplicity of DNS hosts (my own, where many of my domains are registered, and Google where others are registered). The script pauses for each DNS change required, to give me time to go to the appropriate DNS host, make the changes, and give them a chance to propagate. The script then creates a tarball with all the certs ready to be distributed.

You should have a directory in /etc/certs/ for each domain you wish to renew.

#! /bin/bash

for i in /etc/certs/*
	i=`echo $i | sed -re 's/.*\/(.*)/\1/'`
	echo '-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-'
	echo "        $i         *.$i"
	echo '-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-'
	sudo ./certbot-auto certonly --manual \
	--server \
	--preferred-challenges dns -d "$i,*.$i"

echo '-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-'
echo creating all certs tarball
echo '-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-.,,.-*"*-'
sudo tar -czhvf certs.tgz  --xform 's|^etc/letsencrypt/live/\(.*\)|\1|' /etc/letsencrypt/live/*

Distributing the Certs

Once the tarball is ready, I run this next script from my laptop. It fetches the tarball from the cert server, then uploads it to each server. It then extracts the certs into the /etc/certs directory (which must already exist with proper permissions), performs cleanup (including deleting any certs not needed by that server), and restarts any services that might be affected by the change. I’ve edited it to only show two servers; more would just be tedium.

You really want password-free (public key) logins to your servers or you’ll be typing a lot of passwords.

#! /bin/bash

# You should have executed 'sudo ./renewwildcards' on straighteight

# Copy the cert tarballs here
echo " <---------------" 
scp ~/certs/ 
# Distribute the cert tarballs to the various hosts 
# Need to be local to the sacdoc home intranet for many of these 
# modify the hosts' sudoer files with something like: 
# TLS certificate distribution 
# %adm ALL=(ALL) NOPASSWD: /bin/tar xzf /home/twoprops/certs.tgz 
# %adm ALL=(ALL) NOPASSWD: /usr/sbin/service webmin stop 
# %adm ALL=(ALL) NOPASSWD: /usr/sbin/service webmin start 
# %adm ALL=(ALL) NOPASSWD: /usr/sbin/service apache2 reload 
# to allow a non-interactive expansion of the certs and reloading 
# the affected services 
# If you don't have time to set up the sudoers file, adding the '-t' 
# switch to the ssh command invoking sudo will allow interactive 
# password entry 
# I have a different username on different hosts, so it has to be 
# specified on a per-host basis 

#### costco #### 
path=/home/$user echo echo "---------------> $host"
scp ~/certs/certs.tgz $user@$host:$path/certs.tgz
ssh $user@$host "cd /etc/certs; sudo /bin/tar xzf $path/certs.tgz"
ssh $user@$host rm $path/certs.tgz
ssh $user@$host sudo /usr/sbin/service apache2 reload

#### fiona ####

echo "---------------> $host"
scp ~/certs/certs.tgz $user@$host:$path/certs.tgz
ssh $user@$host "cd /etc/certs; sudo /bin/tar xzf $path/certs.tgz"
ssh $user@$host rm $path/certs.tgz
ssh $user@$host sudo /usr/sbin/service webmin stop
ssh $user@$host sudo /usr/sbin/service webmin start