Location to store ipset blocklist?

[john8791]

I currently run DD-WRT (Kongac 26190M) on a Netgear AC1450. I have had good luck with my setup, except DD-WRT iptables does not support ipset even with Entware installed. I use a script to block all TOR nodes from my kids which cuts my internet speed by 1/3. I tuned iptables to make the TOR rule only apply to their devices but I want a better solution.

I am considering buying a used Asus RT-AC68U and running Merlin. I found this article describing how to use a script to block TOR exit nodes with ipset, but it says to download the list to "/jffs/ipset_lists". As this list is quite large, and needs to be updated daily or weekly to keep up, wouldn't that wear out the flash memory? I could put it on a USB stick, but this script would execute before the USB was available on boot. Any ideas?

 

[ASAT]

Any ideas?

You could write a "smart update to jffs" feature. You will probably need to diff two sorted files, since there's no timestamp on the downloaded file. Here's a sample code from my "ipset-save" script. I backup my ipset every hour to jffs only if there's been a change. And I prevent backup of empty ipset.

if [ -f "$IPSET_BACKUP" ]; then
  diff -a "$IPSET_BACKUP_TMP" "$IPSET_BACKUP" > /dev/null 2>&1
  if [ $? -eq 0 ] || [ $(wc -c $IPSET_BACKUP_TMP|cut -f1 -d' ') -le 1 ]; then
    # same file, or new file is empty
    rm -f "$IPSET_BACKUP_TMP"
  else
    # different file
    mv -f "$IPSET_BACKUP_TMP" "$IPSET_BACKUP"
  fi
else
  # new file
  mv -f "$IPSET_BACKUP_TMP" "$IPSET_BACKUP"
fi

 

[john8791]

You could write a "smart update to jffs" feature. You will probably need to diff two sorted files, since there's no timestamp on the downloaded file. Here's a sample code from my "ipset-save" script. I backup my ipset every hour to jffs only if there's been a change. And I prevent backup of empty ipset.

if [ -f "$IPSET_BACKUP" ]; then
  diff -a "$IPSET_BACKUP_TMP" "$IPSET_BACKUP" > /dev/null 2>&1
  if [ $? -eq 0 ] || [ $(wc -c $IPSET_BACKUP_TMP|cut -f1 -d' ') -le 1 ]; then
    # same file, or new file is empty
    rm -f "$IPSET_BACKUP_TMP"
  else
    # different file
    mv -f "$IPSET_BACKUP_TMP" "$IPSET_BACKUP"
  fi
else
  # new file
  mv -f "$IPSET_BACKUP_TMP" "$IPSET_BACKUP"
fi

I think I understand. So you set a cron job to copy $IPSET_BACKUP to /jffs? How often is "too often" when writing to a modern router like the RT-AC68U?

 

[ASAT]

How often is "too often" when writing to a modern router like the RT-AC68U?

On the Cron interval, you download the file to /tmp. You can do it as often as you like because it's RAM. Then, do a diff on the /tmp file versus the /jffs file. Be sure to sort the file after downloading, to give it a consistent form, prior to diff'ing. Then, update /jffs only when a change is detected.

Example: download, sort and save to /tmp

wget -O - http://website.com/blocked-ips.txt | sort -n > /tmp/blocked-ips.txt

 

[john8791]

On the Cron interval, you download the file to /tmp. You can do it as often as you like because it's RAM. Then, do a diff on the /tmp file versus the /jffs file. Be sure to sort the file after downloading, to give it a consistent form, prior to diff'ing. Then, update /jffs only when a change is detected.

Example: download, sort and save to /tmp

wget -O - http://website.com/blocked-ips.txt | sort -n > /tmp/blocked-ips.txt

That makes perfect sense now. I didn't think about using the /tmp RAM space. Thanks!

 

[ASAT]

Alternatively, when a file is generated locally (i.e. not downloaded) you may compare the timestamps of the files to know when there is change. Diff'ing takes more resources and the files must be sorted. So, this technique here is much more efficient for when a file is generated locally. In this case here, an updated timestamp is the change.

Example: Copy /tmp file to /jffs only when there is timestamp change

# save ntp drift file to jffs
NTPDRIFT_TMP=/tmp/etc/ntp/ntp.drift
NTPDRIFT_JFFS=/jffs/ntp/ntp.drift
if [ -f "$NTPDRIFT_TMP" ]; then
  if [ ! -f "$NTPDRIFT_JFFS" ] || [ $(date -r "$NTPDRIFT_TMP" +%s) -gt $(date -r "$NTPDRIFT_JFFS" +%s) ];  then
    cp -p "$NTPDRIFT_TMP" "$NTPDRIFT_JFFS"
  fi
fi