Yesterday, Google released Chrome 90, and with that "HTTPS" is becoming the default protocol if you enter just a hostname into the URL bar without specifying the protocol [1]. This is the latest indication that the EFF's "HTTPS Everywhere" initiative is succeeding [2][3]. Browsers are more and more likely to push users to encrypted content. While I applaud this trend, it does have a downside for small internal sites that often make it difficult to configure proper certificates. In addition, browsers are becoming pickier as to what certificates they accept. For example, in the "good old days", I could set up internal certificates that were valid for 10 years, not having to worry about the expiring. Currently, browsers will reject certificates valid for more than 13 months (398 days) [4]. 

Luckily, there is a solution: The "ACME" protocol popularized by the Let's Encrypt initiative makes it relatively painless to renew certificates. Sadly, not all software supports it, and in particular, IoT devices often do not support it [5].

Why Run Your Own Certificate Authority

Let's step back for a moment, and look at the certificate authorities. Why do you want to run your own? There are a couple of reasons that made me use my own certificate authority:

1. Privacy: Public certificate authorities maintain certificate transparency logs. These logs are made public and are easily searchable. I do not want my internal hostnames to show up in these logs [6].
2. Flexibility: Sometimes, I do not want to play by the rules that public certificate authorities have to play by. I do still have a pretty nice security camera that I don't want to toss that only supports 1024 bit private keys. Verifying an internal hostname can also be difficult if you are using a nonpublic top-level domain, or if the host is not reachable for certificate validation (you will need to use DNS which requires a cooperating DNS host).

How to Get Started With Your Private Certificate Authority

I found the easiest way to set up your own certificate authority (and be able to use the ACME protocol) is smallstep [7]. Smallstep is often used for SSH keys, but it is also a very capable certificate authority and easily runs in a virtual machine or container. When I started to use smallstep, it required a bit of work (a patch) to be compatible with macOS. But I believe this issue has been fixed yet. Certificates obtained via ACME were missing the "CommonName" that MacOS (and the RFC) require. Today, the "Getting Started Guide" is all you should need.

The setup process will do all the hard work for you. You will get a CA certificate, and Intermediate certificate and should be ready to go in no time. Just make sure to import the CA certificate into your clients and trust them. (I include the intermediate certificate as well to avoid some issues with the intermediate certificate not being included by a server).

The certificate authority doesn't necessarily have to be online all the time, but for ACME to work best and for your systems to be able to automatically renew certificates, you may just want to keep it running.

Using Your Own Certificate Authority with "certbot"

"certbot" is the most popular ACME client these days. All you need to do to use it with smallstep is to point it at your own smallstep server:

certbot certonly -d example.com --server https://internal-ca-hostname:8443/acme/acme/directory

by default, smallstep listens on port 8443. The system you run certbot on needs to trust the smallstep CA or the connection will fail.

For internal verification, I also like DNS instead of the normal default HTTP. You often deal with devices that have odd web server configurations. So you can not easily spin up a stand-alone web server, or use the nginx/apache plugins. The home directory is also not always writeable (or even present). So DNS makes for a nice alternative. To use DNS, it is easiest if you run an internal authoritative DNS server for the respective zone, and enable dynamic updates. Certbot has a "dns-rfc2136" module that supports authenticated dynamic DNS updates. [9]

A Lot of Moving Parts...

So here is a quick "To Do" list of everything you need in the rough order you should set it up:

1. Register a domain for internal use (or use a subdomain of one you already own). Do NOT use .local internally.
2. Setup an internal authoritative DNS server
3. Enable authenticated dynamic DNS on that DNS server and allow updates from your internal IPs using specific keys.
4. Install smallstep
5. Install certbot and the rfc2136 module
6. Run certbot to get your new certificates
7. Symlink the certificates to the location where your system expects them
8. Use certbot renewal hooks to do additional operations on certificates as needed (e.g. if you need to create Java keystores or restart services)

Enjoy!

[1] https://blog.chromium.org/2021/03/a-safer-default-for-navigation-https.html
[2] https://www.eff.org/https-everywhere
[3] https://transparencyreport.google.com/https/overview?hl=en
[4] https://support.apple.com/en-us/HT211025
[5] https://tools.ietf.org/html/rfc8555
[6] https://transparencyreport.google.com/https/certificates?hl=en
[7] https://smallstep.com/docs/step-ca
[8] https://eff.org/certbot
[9] https://tools.ietf.org/html/rfc2136

---
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS.edu
Twitter|