wgclient-start with iptables nat rules after router reboot

[zjarkov]

I have a wgclient-start (and its wgclient-stop counterpart) with some iptables rules to modify a nat. It works as expected when enabling/disabling wg client. However, after router (RT-AX86U with 388.2_2) reboot, all iptables rules set by the script are missing though the script has been obviously launched - I can see traces in syslog and also has put touch /tmp/wgclient$1-start into wgclient-start for sure.
Does anybody know some workaround or how can I trace what is wrong?

 

[Maverickcdn]

My guess... the firewall gets reloaded multiple times on startup, what Im thinking is wgclient-start is run at boot and later on during boot the firewall gets reloaded erasing your rules... check in syslog for May 4 23:05:11 MyROUTERname rc_service: waitting "restart_firewall" via ... appearing after wgclient-start has run

 

[zjarkov]

My guess... the firewall gets reloaded multiple times on startup, what Im thinking is wgclient-start is run at boot and later on during boot the firewall gets reloaded erasing your rules... check in syslog for May 4 23:05:11 MyROUTERname rc_service: waitting "restart_firewall" via ... appearing after wgclient-start has run

Thrilled for a while, but this doesn't seems to be the case :-(

@router:/tmp/home/root# grep -ni restart_firewall /tmp/syslog.log*
/tmp/syslog.log:428:May  5 07:05:10 rc_service: service 1511:notify_rc restart_firewall
/tmp/syslog.log:541:May  5 07:05:13 rc_service: waitting "restart_firewall" via  ...
@router:/tmp/home/root# grep -ni wgclient /tmp/syslog.log*
/tmp/syslog.log:591:Nov 25 02:19:59 custom_script: Running /jffs/scripts/wgclient-start (args: 1)

 

[Maverickcdn]

bummer, was a wild a** guess.

Someone else might have a better idea.

 

[ZebMcKayhan]

Thrilled for a while, but this doesn't seems to be the case :-(

@router:/tmp/home/root# grep -ni restart_firewall /tmp/syslog.log*
/tmp/syslog.log:428:May  5 07:05:10 rc_service: service 1511:notify_rc restart_firewall
/tmp/syslog.log:541:May  5 07:05:13 rc_service: waitting "restart_firewall" via  ...
@router:/tmp/home/root# grep -ni wgclient /tmp/syslog.log*
/tmp/syslog.log:591:Nov 25 02:19:59 custom_script: Running /jffs/scripts/wgclient-start (args: 1)

I think @Maverickcdn is correct anyway. I see 3 different ways of solving this
1. Put your nat rules in nat-start instead.
2. Restarting wg peers in nat-start
3. Fw uses /etc/wg/fw_wgc1_nat.sh to store these rules, you could pipe them there. This would be elegant but may require some tinkering, but I could help you.

 

[zjarkov]

bummer, was a wild a** guess.

Someone else might have a better idea.

Hi, your guess is correct at the end, just traces in syslog are not as originally expected, thx for a tip.

I let log file to be polluted from nat-start and wgclient-start and (at least) in my case the nat-start is called 3 times during boot when the last call is done approx 13s after WG start script.

 

[zjarkov]

I think @Maverickcdn is correct anyway. I see 3 different ways of solving this
1. Put your nat rules in nat-start instead.
2. Restarting wg peers in nat-start
3. Fw uses /etc/wg/fw_wgc1_nat.sh to store these rules, you could pipe them there. This would be elegant but may require some tinkering, but I could help you.

Thank you for the options.
I have a few questions to 3)

• Does it persist between FW upgrades?
• Under what condition it is (re)generated? I presume I could loose customization from time to time.

 

[ZebMcKayhan]

Does it persist between FW upgrades?

This is the tinkering needed, from the date mark on my file I would say it's created when you hit apply in the wg client gui section (so, when you create or change something). I wouldn't think it's changed at fw update but I don't know. I haven't updated since I created mine.

Under what condition it is (re)generated? I presume I could loose customization from time to time.

I don't know if there are any other reasons for recreation.

So, to be safe, maybe your wgclient-start should check some #tag you put there and if it does not exist you need to re-add them. Wgclient-start probably also need to execute them.

 

[zjarkov]

At the end I have used nat-start, as suggested by @ZebMcKayhan, to check for running wg clients and rerun wgclient-start which has been made safe to reload/rerun - avoiding adding duplicate nat rules.

 

[ZebMcKayhan]

At the end I have used nat-start, as suggested by @ZebMcKayhan, to check for running wg clients and rerun wgclient-start which has been made safe to reload/rerun - avoiding adding duplicit nat rules.

Would you mindre sharing your nat-start code part that does this so others could benefit?

 

[zjarkov]

Would you mindre sharing your nat-start code part that does this so others could benefit?

Sure, anyway, I'll be thankful if you find any problem or possible vulnerability...
Here you go:

nat-start

#!/bin/sh

# cycle all running WireGuard clients
for wgc in `ls /tmp/wgclient-enabled_* 2>/dev/null`; do
    # reload/re-run the `wgclient-start X`
    /jffs/scripts/wgclient-start ${wgc/*_/}
done

wgclient-start

#!/bin/sh

# set NAT rules for WireGuard client
/jffs/scripts/wgclient-natrules $1 I
# mark that WG client start script has been run
touch /tmp/wgclient-enabled_$1

wgclient-stop

#!/bin/sh

# remove NAT rules for WireGuard client
/jffs/scripts/wgclient-natrules $1 D
# remove mark that WG client start script has been run
rm -f /tmp/wgclient-enabled_$1

wgclient-natrules

#!/bin/sh

USAGE=$(cat << EOF
USAGE:
    `basename $0` WGCLIENT_NUM [OPERATION]

        WGCLIENT_NUM    WireGuard client number, the same as passed
                        to 'wgclient-start' and 'wgclient-stop' scripts.
        OPERATION       Operation to be performed, valid values are 'I'
                        (insert) or 'D' (delete). Optional, default 'I'.
EOF
)

# check input arguments
OP="${2:-I}"
[ 'I' == $OP -o 'D' == $OP ] || { echo -e "Invalid OPERATION '$OP'\n$USAGE" >&2; exit 1; }

# helper to process the nat rule
natrule()
{
    if [ 'I' == $OP ]; then
        iptables -t nat -C $@ 2>/dev/null || iptables -t nat -I $@
    else
        iptables -t nat -D $@
    fi
}

# rules for WireGuard client 1
if [ 1 -eq $1 ]; then
    # manage custom NAT rules here
    natrule PREROUTING -i wgc$1 -p tcp --dport 55443 -s 10.8.50.0/28 -j DNAT --to-destination 192.168.0.15:443
    natrule POSTROUTING -o br0 -p tcp -s 10.8.50.0/28 -d 192.168.0.15 --dport 443 -j MASQUERADE

    # ...
fi

 

[Maverickcdn]

Sure, anyway, I'll be thankful if you find any problem or possible vulnerability...
wgclient-natrules

# check input arguments
[ 'I' == $OP -o 'D' == $OP ] || { echo -e "Invalid OPERATION '$OP'\n$USAGE"

Missing a closing brace } on your echo or its an erroneous {

Otherwise glad we were able to lead you in the right direction!

 

[zjarkov]

Missing a closing brace } on your echo or its an erroneous {

Otherwise glad we were able to lead you in the right direction!

Argh, copy pasted from console and missed that overflowed one
fixed...

 

[ZebMcKayhan]

@zjarkov Thanks for sharing! Nice way to setup your rules in a single line like this!

I couldnt keep my hands off, so I borrowed your rules and made this, mostly because I'm curious if it works. It does the job but I don't have the ability to test against the firmware right now:

wgclient-start

#!/bin/sh

file="/etc/wg/fw_wgc1_nat.sh"

cat <<EOF | while read -r NAT_RULE;
iptables -t nat -I PREROUTING -i wgc1 -p tcp --dport 55443 -s 10.8.50.0/28 -j DNAT --to-destination 192.168.0.15:443
iptables -t nat -I POSTROUTING -o br0 -p tcp -s 10.8.50.0/28 -d 192.168.0.15 --dport 443 -j MASQUERADE
EOF

do
if [ -z "$(cat "$file" | grep "$NAT_RULE")" ];then #if rule is not found in files
   echo $NAT_RULE >> $file #append rule to file
fi
   eval $NAT_RULE #Execute rule
done

wgclient-stop

#!/bin/sh
iptables -t nat -D PREROUTING -i wgc1 -p tcp --dport 55443 -s 10.8.50.0/28 -j DNAT --to-destination 192.168.0.15:443 
iptables -t nat -D POSTROUTING -o br0 -p tcp -s 10.8.50.0/28 -d 192.168.0.15 --dport 443 -j MASQUERADE

I'm not a coder so feel free to criticize.

It will look if the rules are in the fw file and add them if they are not there. Then execute them.
but if you change the rules it will not delete any entries in the fw file and I don't know when this file is recreated, possibly never, so one needs to check this when updating the rules.

 

[zjarkov]

@zjarkov Thanks for sharing! Nice way to setup your rules in a single line like this!

I couldnt keep my hands off, so I borrowed your rules and made this, mostly because I'm curious if it works. It does the job but I don't have the ability to test against the firmware right now:

wgclient-start

#!/bin/sh

file="/etc/wg/fw_wgc1_nat.sh"

cat <<EOF | while read -r NAT_RULE;
iptables -t nat -I PREROUTING -i wgc1 -p tcp --dport 55443 -s 10.8.50.0/28 -j DNAT --to-destination 192.168.0.15:443
iptables -t nat -I POSTROUTING -o br0 -p tcp -s 10.8.50.0/28 -d 192.168.0.15 --dport 443 -j MASQUERADE
EOF

do
if [ -z "$(cat "$file" | grep "$NAT_RULE")" ];then #if rule is not found in files
   echo $NAT_RULE >> $file #append rule to file
fi
   eval $NAT_RULE #Execute rule
done

wgclient-stop

#!/bin/sh
iptables -t nat -D PREROUTING -i wgc1 -p tcp --dport 55443 -s 10.8.50.0/28 -j DNAT --to-destination 192.168.0.15:443 
iptables -t nat -D POSTROUTING -o br0 -p tcp -s 10.8.50.0/28 -d 192.168.0.15 --dport 443 -j MASQUERADE

I'm not a coder so feel free to criticize.

It will look if the rules are in the fw file and add them if they are not there. Then execute them.
but if you change the rules it will not delete any entries in the fw file and I don't know when this file is recreated, possibly never, so one needs to check this when updating the rules.

Didn't try it yet but in general that could work with some adjustments.
At first you can do the test more elegant way if grep -q "$NAT_RULE" $file; then and as second you should remove particular rules from /etc/wg/fw_wgc1_nat.sh in wgclient-stop, otherwise, when you stop/disable the client, the rules will persist and 'll be applied after router reboot.

 

[ZebMcKayhan]

At first you can do the test more elegant way if grep -q "$NAT_RULE" $file; then

thanks, Ill make some quick test this evening.

second you should remove particular rules from /etc/wg/fw_wgc1_nat.sh in wgclient-stop, otherwise, when you stop/disable the client, the rules will persist and 'll be applied after router reboot.

I thought this was taken care of in the fw part handling nat events, but I just checked the code and it doesnt. I'm not sure, but it looks like the fw might be changing -I to -D in this file using sed when the peer is stopped, but that would apply to our rules as well. it then looks like the file is deleted after it has been executed. maybe this takes care of both removing our rules and preventing further execution:

snprintf(path, sizeof(path), "%s/fw_%s_nat.sh", WG_DIR_CONF, ifname);
if(f_exists(path)) {
eval("sed", "-i", "s/-I/-D/", path);
eval("sed", "-i", "s/-A/-D/", path);
eval(path);
unlink(path);

the file would need to be re-created when peer is started, but wonder what happens during reboot...

maybe you are better at reading the fw code than me: https://github.com/RMerl/asuswrt-merlin.ng/blob/master/release/src/router/rc/wireguard.c

 

[ZebMcKayhan]

Didn't try it yet but in general that could work with some adjustments.
At first you can do the test more elegant way if grep -q "$NAT_RULE" $file; then and as second you should remove particular rules from /etc/wg/fw_wgc1_nat.sh in wgclient-stop, otherwise, when you stop/disable the client, the rules will persist and 'll be applied after router reboot.

I can confirm that firmware removes ALL rules added to this file, so it's not needed to remove any rules in wgclient-stop. The file is also removed when the client is stopped. I have not checked reboot yet or if all is executed OK at nat-flush event.

I changed the grep statement according to your suggestion and added to only apply on the right peer start. This works for me:
wgclient-start

#!/bin/sh

if [ "$1" -eq "2" ]; then # only for wgc2
   file="/etc/wg/fw_wgc2_nat.sh"
cat <<EOF | while read -r NAT_RULE;
iptables -t nat -I PREROUTING -i wgc2 -p tcp --dport 55443 -s 10.8.50.0/28 -j DNAT --to-destination 192.168.0.15:443
iptables -t nat -I POSTROUTING -o br0 -p tcp -s 10.8.50.0/28 -d 192.168.0.15 --dport 443 -j MASQUERADE
EOF
   do
      if ! grep -q "$NAT_RULE" $file; then
         echo $NAT_RULE >> $file
      fi
         eval $NAT_RULE
   done
fi

wgclient-stop
not needed

I will test reboots tomorrow but I don't expect there to be any issues.

 

[zjarkov]

wgclient-start

#!/bin/sh

if [ "$1" -eq "2" ]; then # only for wgc2
   file="/etc/wg/fw_wgc2_nat.sh"
cat <<EOF | while read -r NAT_RULE;
iptables -t nat -I PREROUTING -i wgc2 -p tcp --dport 55443 -s 10.8.50.0/28 -j DNAT --to-destination 192.168.0.15:443
iptables -t nat -D POSTROUTING -o br0 -p tcp -s 10.8.50.0/28 -d 192.168.0.15 --dport 443 -j MASQUERADE
EOF
   do
      if ! grep -q "$NAT_RULE" $file; then
         echo $NAT_RULE >> $file
      fi
         eval $NAT_RULE
   done
fi

wgclient-stop
not needed

This actually could be nice all-in-one script

Questions:

1. Are /etc/wg/* files anyhow documented and stable (naming, existence, its purpose, whatever...)? I am a "normal" user who usually fiddle only with official APIs to avoid future problems (unannounced change of background stuff).
2. I might be wrong but IMHO this way it is rather wgclient%d.postconf than wgclient-start script. Of course it would need a slight modification to be truly postconf.

 

[ZebMcKayhan]

Are /etc/wg/* files anyhow documented and stable (naming, existence, its purpose, whatever...)? I am a "normal" user who usually fiddle only with official APIs to avoid future problems (unannounced change of background stuff).

No, they are not official in that sense. I guess a major change to the code could change their names or behaviour. Possibly the 3.0.0.6 release would render this useless (together with many add-ons).

I might be wrong but IMHO this way it is rather wgclient%d.postconf than wgclient-start script. Of course it would need a slight modification to be truly postconf.

Well, in the sense as we altering config files, but wg postconf is executed after /etc/wg/wgcX.conf is created and before it is sent to the interface. Our nat file wouldn't exist then.

 

[ZebMcKayhan]

This actually could be nice all-in-one script

I could confirm that it does the job through ordinary reboot. I wouldn't expect accidental reboot to be any difference.
And, as you say, nat rules gets flushed after wgc start and the rules are still there so it will be applied during nat-flush events as well.

Of course the use of this file is unofficial, so maybe this is not the way to go, but I leave it here for anyone to use if they wish. I wouldn't expect this to change in the coming minor releases but who knows for the next major release.