Tutorial Installing Caddy reverse proxy

[spindrift]

July 14, 2024: Edited this post to reflect working instructions for current version of caddy (2.8.4).

---

Hi there! This is my first post, and SNB has been massively helpful in my decision to get an ASUS router (AX86U) as well as integrating it into my home network.
I know that nginx is available as a Merlin add-on, but I've become quite smitten with caddy as it automatically generates certs and I generally find the config syntax a lot easier.
I wanted to see if I could install caddy on my router, and surprisingly, it worked! Here are the steps I used for anyone else interested. (If you see anything incorrect or inadvisable, please do let me know, though I only ask you do so constructively!)

Assumptions/Requirements

Requirement	Rationale
JFFS is on and persistent	If you want to keep Go on your system, you'll need to export variables in /jffs/scripts/init-start.
USB drive is mounted	tmpfs will run out of memory if we try to build XCaddy inside it.
EntWare is installed on USB at /opt	We'll be using the EntWare folder to store Caddy, and the instructions assume the default install path of /opt.
Web Access from WAN is disabled	Caddy needs to be the only thing listening for web connections over WAN.
Local Access Config has ports changed away from 80/443	Not necessary if you want Caddy to use different ports.
Dynamic DNS enabled	Any reverse proxy requires a way for WAN traffic to be forwarded to it.
Port Forwarding set to forward 80/443 to router	(Or whatever ports you want Caddy to accept connections on.)

All steps below assume you're logged into the router over SSH.

Install Go

opkg update
opkg install go

Folders and variables: Go
By default, Go sets itself up in $HOME, which by default is /root on tmpfs. We need to relocate it to the USB drive, and set environment variables so the Go build system knows where to find itself and its build system.

# Below line does not seem relevant anymore, but keeping it in case it is on your router.
mv $HOME/go /opt/home # So we don't run out of space
export GOROOT=/opt/bin/go # Go is here
export GOPATH=/opt/home/go # XCaddy will go here
export GOPROXY="https://proxy.golang.org" # Needed for install
export GOSUMDB="sum.golang.org" # Needed for install
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin # So we can access both
export GOTMPDIR=/opt/tmp # Temporary build folder for Go (change export to TMPDIR if you have issues)

Folders and variables: Caddy
Caddy uses $XDG_DATA_HOME and $XDG_CONFIG_HOME to set its own environment. We want to keep it on /opt, and make sure it knows that.

mkdir /opt/var/log/caddy # Caddy logs will go here (need to be referenced in Caddyfile)
mkdir /opt/share/caddy # Caddy certs will go here
mkdir /opt/etc/caddy # Caddyfile will go here
# This sets Caddy's root
cat > /opt/etc/caddy/Caddyfile << EOF
{
  storage file_system {
    root /opt/share/caddy
  }
}
EOF

export XDG_DATA_HOME=/opt/share
export XDG_CONFIG_HOME=/opt/etc

Installing XCaddy and building Caddy

Replace with your own desired plugins in the demonstrated format, or none. Only --output is required.
If you want Caddy to generate certs automatically, you DO need a plugin for a supported DNS handler (like cloudflare).
transform-encoder is required to make Caddy output logs in common_log format needed by Fail2Ban etc.

go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest

# Build Caddy
xcaddy build --output /opt/bin \
--with github.com/caddy-dns/cloudflare \
--with github.com/caddyserver/transform-encoder

# Clean Go build cache
go clean -cache
go clean -modcache

NOTE: On my router, I got a fatal error on the very last step, claiming /opt/bin: permission denied. Nonetheless, the Caddy binary did compile to the right spot, and everything continued to work fine. If you know how to subvert this error, please let me know.

Installing as a service

We'll use the EntWare init script format to let us easily run Caddy in the background, restart, and check on it.

cat > /opt/etc/init.d/S98caddy << EOF
#!/bin/sh

ENABLED="yes"
PROCS="caddy"
ARGS="start --config $XDG_CONFIG_HOME/caddy/Caddyfile"
WORK_DIR="$XDG_DATA_HOME/caddy"
DESC=$PROCS
PREARGS=""
PRECMD=""
POSTCMD=""

. /opt/etc/init.d/rc.func
EOF

chmod +x /opt/etc/init.d/S98caddy # Make it executable

Persisting variables

NOTE: Caddy (and I assume any other Go app) will try to retrieve the number of CPU cores by querying /proc/self/cgroup—which doesn't exist on my AX86U—and will throw a (non-fatal) warning when it can't. You can get around this by exporting GOMAXPROCS yourself. I don't fully know what Go or Caddy does with this information, or what happens if you get it wrong. I left it commented so you can decide to set it yourself. (At any rate, the warning isn't visible when you run Caddy in the background as we're doing.)

cat >> /jffs/scripts/init-start << EOF
# For Caddy
export XDG_DATA_HOME=/opt/share
export XDG_CONFIG_HOME=/opt/etc
# export GOMAXPROCS=4 # Number of cores your CPU has

# Only if you want to keep the Go build system around
export GOROOT=/opt/bin/go
export GOPATH=/opt/home/go
export GOPROXY="https://proxy.golang.org"
export GOSUMDB="sum.golang.org"
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin
export GOTMPDIR=/opt/tmp # Change GOTMPDIR to TMPDIR if you have issues
EOF

Starting Caddy

Before you start it, you'll want to make sure you build your Caddyfile and check it with caddy fmt (while in the same folder as the Caddyfile). If it's not formatted right, you can fix it in-place by running caddy fmt --overwrite.

/opt/etc/init.d/S98caddy start # Start Caddy in the background
/opt/etc/init.d/S98caddy check # Verify Caddy is running (if it's not, you probably have a config issue)

Troubleshooting

caddy start runs Caddy in the background, which keeps us from seeing its runtime logs (which only seem to show up on stdout).
If Caddy is dying unexpectedly, or something otherwise seems off, run /opt/etc/init.d/S98caddy stop (or killall caddy) to quit the service, then manually run caddy run --config /opt/etc/caddy/Caddyfile to see what it's doing. When you're satisfied, you can ^C and go back to running it through the init script.

If Caddy isn't working and caddy fmt says your config file is off, but it looks okay, try checking for random spaces around the {} brackets and deleting them. (Or use caddy fmt --overwrite)

Example Caddyfile

Just for fun, here's an example Caddyfile with DNS settings, logging, and a couple reverse proxy entries, one of which has CORS set and the other does not.
It's recommended to use wildcard domains for reverse proxying, so you don't leak your servers to SSL registrars, although sometimes that also means you have to set up CORS for things like server dashboards that use IFrames.
Also note that whatever you're using for your DNS, you have to set up a CNAME entry that points to your DDNS domain for every subdomain you wish to reverse proxy.

{
        storage file_system {
                root /opt/share/caddy
        }
}
*.domain.com {
        tls {
                dns cloudflare <cloudflare DNS key>
        }
        log {
                output file /opt/var/log/caddy/<domain>-access.log
                format transform "{common_log}"
        }
        @app host app.domain.com
        handle @app {
                reverse_proxy <server IP>:<port> {
                        header_down Access-Control-Allow-Origin *
                }
        }
        @another_app host anotherapp.domain.com
        handle @another_app {
                reverse_proxy <server IP>:<port>
        }
}

Editing Caddyfile remotely via SFTP

I don't want to SSH into my router and use vi every time I need to update my Caddyfile. If you don't have a preferred way to edit router files remotely yourself, I can at least recommend the SFTP sync extension for VS Code, the configuration of which is outside the scope of this post. However, it does require an SFTP server be running, so if you want to go this route make sure you also:

opkg update
opkg install openssh-sftp-server

(Open to feedback if there are better ways to do this.)

Addendum for older models

See what hexcallm had to do additionally for their AC88U:

Tutorial - Installing Caddy reverse proxy

July 14, 2024: Edited this post to reflect working instructions for current version of caddy (2.8.4). Hi there! This is my first post, and SNB has been massively helpful in my decision to get an ASUS router (AX86U) as well as integrating it into my home network. I know that nginx is available...

www.snbforums.com

 

[Jeffrey Young]

Welcome to the forums and thank for your contribution.

 

[L&LD]

Welcome to the forums @spindrift.

A kind thank you from me as well. Looking eagerly to your future contributions here too!

 

[hexcallm]

Just wanted to add a tip in case anyone else is hung up by this. If you have an older model (AC88U in my case), you may need to install go_nohf:

opkg install go_nohf

Try it if you're getting "illegal instruction" when running go.

Edit:

Of course I immediately hit another compatibility issue. The current @latest version of xcaddy (v0.3.5 from August 2023) is not compatible with the version of go (1.20.7) that is available in the entware go_nohf package. This is due to Go 1.21 (released August 2023) introducing the 'toolchain' directive, which caddy appears to use. After a bumbling attempt to builddirectly using go, I tried an older version of xcaddy.

To make it work, use a prior version tag in the command outined by spindrift above to install xcaddy.

go install github.com/caddyserver/xcaddy/cmd/xcaddy@v0.3.4

In my case, v0.3.4 worked (of course it took 1.5 hours to build and interrupted my connectivity--makes me think I need an armv5 container/vm running on a more robust machine if I have to do this again). For reference, the xcaddy releases page: https://github.com/caddyserver/xcaddy/releases

PS, thanks to spindrift. (And side note, I also enjoy your sparkling water.)

 

[spindrift]

Just a little update, since I keep referring to my own forum post whenever I need to do something with Caddy

If you've already installed Caddy and you just want to add a module or update to the latest version, then at the time of writing these are the necessary steps (at least for my RT-AX86U):

# Update Go
opkg update
opkg upgrade go

# Re-export necessary environment variables
export GOROOT=/opt/bin/go # Go is here
export GOPATH=/opt/home/go # XCaddy will go here
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin # So we can access both
export GOTMPDIR=/opt/tmp # Temporary build folder for Go
export XDG_DATA_HOME=/opt/share
export XDG_CONFIG_HOME=/opt/etc

# Needed for latest Go (1.21) at time of writing
export GOPROXY="https://proxy.golang.org"
export GOSUMDB="sum.golang.org"

# I needed this for admittedly unclear reasons; add it if Caddy complains
# export GOMAXPROCS=4 # Number of cores your CPU has

# Update XCaddy
go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest

# Build Caddy with desired plugins (this is my personal list)
xcaddy build --output /opt/bin \
    --with github.com/caddy-dns/cloudflare \
    --with github.com/WeidiDeng/caddy-cloudflare-ip \
    --with github.com/lanrat/caddy-dynamic-remoteip \
    --with github.com/caddyserver/transform-encoder \
    --with github.com/caddyserver/replace-response

# Clean Go download and build caches
go clean -cache
go clean -modcache

I will note that on this latest build, I DID get a fatal error on the very last step, claiming /opt/bin: permission denied. Nonetheless, the caddy binary did compile to the right spot.

 

[spindrift]

I edited the top-level post with updated instructions given current versions of Caddy, Go, and so on.