hugo-relie/content/en/posts/dns-challenge.md

title	date	draft	toc	images	tags
How to do HTTPS at home (when your infrastructure is private)	2024-07-02T21:00:50+02:00	true	true		self-hosting sysadmin

The problem of having a self-hosted infrastructure

I've been maintaining a personal homelab and self-hosted infrastructure for a few years now, but one of the most infuriating pages when starting such project is this dreaded Warning: Potential Security Risk Ahead page that appears when you're using a self-hosted certificate, or when trying to use a password on a website or app that is served through plain HTTP.

While acceptable if you're alone on your own infrastructure or dev environment, this poses several issues if many other contexts:

• It is not acceptable to publicly expose a website presenting this issue
• It's not advisable to say "hey look, I know that your browser gives you a big red warning, but it's okay, you can just accept" to friends/family/etc. It's just a very bad habit to have
• After a while, it really starts to get on your nerve

Thankfully a free solution for that, which you will probably know already, has existed for almost ten (10) years now: Let's Encrypt and the ACME protocol

{{< callout type="note" >}} I promise this is not yet another Let's Encrypt tutorial, well it is, but for a more specific use-case {{< /callout >}}

The Let's Encrypt solution

What is Let's Encrypt

Let's Encrypt is a nonprofit certificate authority founded in November 2014. Its main goal was to provide an easy and free way to obtain a TLS certificate in order to make it easy to use HTTPS everywhere.

The ACME protocol developed by Let's Encrypt is an automated verification system aiming at doing the following:

• verifying that you own the domain for which you want a certificate
• creating and registering that certificate
• delivering the certificate to you

Most client implementation also have an automated renewal system, further reducing the workload for sysadmins.

The current specification for the ACME protocol proposes two (2) types of challenges to prove ownership and control over a domain: HTTP-01 and DNS-01 challenge.

{{< callout type="note" >}} Actually there are two (2) others: TLS-SNI-01 which is now disabled, and TLS-ALPN-01 which is only aimed at a very specific category of users, which we will ignore here. {{< /callout >}}

The common solution: HTTP challenge

The HTTP-01 challenge is the most common type of ACME challenge, and will satisfy most use-cases.

For this challenge, you need the following elements :

• A domain name and a record for that domain in a public DNS server (it can be a self-hosted DNS server, your providers', etc)
• Access to a server with a public IP that can be publicly reached

When performing this type of challenge, the following happens (in a very simplified way):

1. Your ACME client will ask to start a challenge to the Let's Encrypt API
2. In return, it will get a token
3. It will then either start a standalone server, or edit the configuration for your current web server (nginx, apache, etc) to serve a file containing the token and a fingerprint of your account key.
4. Let's Encrypt will try to resolve your domain test.example.com.
5. If resolution works, then it will check the url http://test.example.com/.well-known/acme-challenge/<TOKEN>, and verify that the file from step 3 is served with the correct content.

If everything works as expected, then the ACME client can download the certificate and key, and you can configure your reverse proxy or server to use this valid certificate, all is well.

{{< callout type="help" >}} Okay, but my app contains my accounts, or my proxmox management interface, and I don't really want to make it public, so how does it work here? {{< /callout >}}

Well it doesn't. For this type of challenge to work, the application server must be public. For this challenge you need to prove that you have control over the application that uses the target domain (even if you don't control the domain itself). But the DNS-01 challenge bypasses this limitation.

When it's not enough: the DNS challenge

As we saw in the previous section, sometimes, for various reasons, your application server is in a private zone. It must be only reachable from inside a private network, but you still want to be able to use a free Let's Encrypt certificate.

For this purpose, the DNS-01 challenge is based on proving that you have control over the DNS server itself, instead of the application server.

For this type of challenge, the following elements are needed :

• A public DNS server you have control over (can be a self-hosted server, or your DNS provider)
• A ACME client (usually it would be on your application server), it doesn't need to be public

Then, the challenge is done the following way :

1. Your ACME client will ask to start a challenge to the Let's Encrypt API.
2. In return, it will get a token.
3. The client then created a TXT record at _acme-challenge.test.example.com derived from the token and your account key.
4. Let's Encrypt will try to resolve the expected TXT record, and verify that the content is correct.

If the verification succeeds, you can download your certificate and key, just like the other type of challenge.

It's important to note that at no point in time did Let's Encrypt have access to the application server itself, because this challenges involves proving that you control the domain, not that you control the destination of that domain.

As someone trying to use a valid certificate for my proxmox interface, this is the way I would want to go, because it would allow me to have a valid certificate, despite my server not being public at all. So let's see how it works in practice.

DNS challenge in practice

For this example, I will try to obtain a certificate for my own domain example.internal.faercol.me.As this name hints, it is an internal domain and should not be publicly reachable, so this means I'm going to use a DNS challenge. I don't really want to use my DNS provider API for this, so I'm going to use a self-hosted bind server for that.

Configuring the DNS server

Enabling remote DNS zone modification

Performing the challenge

Bonus: completely hiding your private domains from outside