Script to query for DNSCrypt resolvers

[Denna]

I'm trying to create a script that:

a) Extracts DNSCrypt resolver names from all filenames beginning with "S09dnscrypt-proxy" in the /mnt/sda1/entware-ng.arm/etc/init.d directory.

b) Performs a lookup for the resolver names' ip address

c) Adds the resolver names' ip addresses to an ipset set called dnscrypt_ips

Currently, there are two S09dnscrypt-proxy files. In the future, there could be more by appending the next number to the end of the base filename "S09dnscrypt-proxy".

S09dnscrypt-proxy

S09dnscrypt-proxy1​

Step 1 - Extract the resolver name in the file "S09dnscrypt-proxy".

Each of the S09dnscrypt-proxy* files contain the following line where the resolver name to be extracted is after the "-R ".

ARGS="--local-address=127.0.0.1:40 --daemonize -R dnscrypt.eu-dk"

In this example, the resolver name is "dnscrypt.eu-dk".​

Step 2 - Lookup the resolver name and get its ip address.

Resolver names are stored in /opt/share/dnscrypt-proxy/dnscrypt-resolvers.csv. There are dozens of lines in this file, each starting with a different resolver name. Below is an example file.

https://github.com/jedisct1/dnscrypt-proxy/blob/master/dnscrypt-resolvers.csv

The column headers in this file are as follows:

Name,"Full name","Description","Location","Coordinates",URL,Version,DNSSEC validation,No logs,Namecoin,Resolver address,Provider name,Provider public key,Provider public key TXT record

In this file, the dnscrypt.eu-dk line is as follows:

dnscrypt.eu-dk,"DNSCrypt.eu Denmark","Free, non-logged, uncensored. Hosted by Netgroup.","Denmark","","https://dnscrypt.eu,1,yes,yes,no,77.66.84.233,2.dnscrypt-cert.resolver2.dnscrypt.eu,3748:5585:E3B9088:FD25:AD36:B037:01F5:520C648:9E9AD52:1457:4955:9F0A:9955,pubkey.resolver2.dnscrypt.eu"
​

The ip address to be extracted is "77.66.84.233".​

Step 3 - Add this ip address to the dnscrypt_ips ipset set.

ipset create DNSCRYPT_IPS hash:ip
ipset add dnscrypt_ips 77.66.84.233

Step 4 - Iterate for additional S09dnscrypt-proxy* files, if any.

I came up with the following partial solution.

1) Create a script with the following commands.

#!/bin/sh
#DNSCRYPT_RES1 variable = Search for the line that begins with "ARGS=" (without quotes) and print the 4th column.
export DNSCRYPT_RES1=$(awk ' /^ARGS=/ {print $4}' /mnt/sda1/entware-ng.arm/etc/init.d/S09dnscrypt-proxy | sed 's/"/,/')

... which returns

dnscrypt.eu-dk,

NOTE: This only works if the syntax of the "ARGS=" line doesn't change. I'd prefer to determine the position of the character after the string "-R " (note the space character) and extract the following consecutive characters up to the next quote OR space character.
​

Lookup the resolver's name in the .CSV file and extract the characters between the 10th and 11th commas.

Iterate for additional files.​

 

[Fitz Mutch]

Grab the IP

/bin/grep -F "ns0.dnscrypt.is" /opt/share/dnscrypt-proxy/dnscrypt-resolvers.csv | /bin/sed "s/\([^\"]*\"\)[^\"]*\(\"[^\"]*\)/\1(he cut me)\2/g" | /usr/bin/cut -d, -f11

Now the IP will always be in the 11th column because the sed command will remove everything between the quotes. This fixes the issue with embedded commas. Not perfect, but it works.

 

[Denna]

@Fitz Mutch,

​

I'm reading up on the sed command and regexp.

The s command is for substituting. What does the rest do ?

sed "s/\([^\"]*\"\)[^\"]*\(\"[^\"]*\)/\1(he cut me)\2/g"

 

[Fitz Mutch]

The s command is for substituting. What does the rest do ?

sed -r 's/pattern/replacement/g'

The pattern string says:

• match a string of characters ending with a double-quote
• then match all characters between the double-quotes and discard them
• then match a second string of characters beginning with a double-quote until the next double-quote or end of line

The replacement string says:

• print the first string of matched characters
• then print an informative message to show where the discarded characters once lived
• then print the second string of matched characters

The "g" means to replace all occurrences.

Sample output from sed command

ns0.dnscrypt.is,"he cut","he cut","he cut",,https://dnscrypt.is,1,yes,yes,no,93.95.228.87,2.dnscrypt-cert.ns0.dnscrypt.is,EE41:6A83:451C:218F:37B2:B736:78C4:999F:7DE6:89D1:31D2:7866:7C8E:A8BB:1C95:B402,pubkey.ns0.dnscrypt.is

Here's a more readable version of the command

/bin/grep -F "ns0.dnscrypt.is" /opt/share/dnscrypt-proxy/dnscrypt-resolvers.csv | /bin/sed -r 's/([^"]*")[^"]*("[^"]*)/\1he cut\2/g' | /usr/bin/cut -d, -f11

 

[Fitz Mutch]

I found another sed command.

HOWTO: remove commas enclosed in double-quotes using the sed command

cat /opt/share/dnscrypt-proxy/dnscrypt-resolvers.csv | sed -r ':a;s/^(([^"]*,?|"[^",]*",?)*"[^",]*),/\1/;ta'

Once you get to this level of complicated sed programming, I imagine most people would have move on and program it using perl instead. However, perl is not included with the Asuswrt firmware.

Trying to understand that nested group pattern, used in the above sed command, it just makes my head explode like a watermelon.

SOURCE: https://unix.stackexchange.com/ques...een-the-quotes-only-in-a-comma-delimited-file

 

[Denna]

@Fitz Mutch,

Thanks for the detailed help !​

The resolver name "dnscrypt.eu-dk," is output to the DNSCRYPT_RES1 variable with the code below:​

#!/bin/sh
#DNSCRYPT_RES1 variable = Search for the line that begins with "ARGS=" (without quotes) and print the 4th column.
export DNSCRYPT_RES1=$(awk ' /^ARGS=/ {print $4}' /mnt/sda1/entware-ng.arm/etc/init.d/S09dnscrypt-proxy | sed 's/"/,/')

The resolver name's IP address is output with the code below:

#Search for the line that begins with "dnscrypt.eu-dk,", filters out quoted commas and prints the 11th column.
grep -F "dnscrypt.eu-dk," /opt/share/dnscrypt-proxy/dnscrypt-resolvers.csv | sed -r ':r; s/("[^",]+),([^",]*)/\1 \2/g; tr; s/"//g' | cut -f 11 -d ','

How do you use the DNSCRYPT_RES1 variable in place of the "dnscrypt.eu-dk," string with grep ?

 

[redhat27]

Maybe something like this?

#!/bin/sh
INIT_SCRIPTS="/opt/etc/init.d/S09dnscrypt-proxy*"
RESOLVERS_CSV="/opt/share/dnscrypt-proxy/dnscrypt-resolvers.csv"

for script in $INIT_SCRIPTS; do
  RESOLVER=$(sed -n "s/\"$//;/^ARGS/s/^.*-R //p" $script) # pick next word after "-R"
  RESOLVER_IP=$(sed -n "/^$RESOLVER,/p" $RESOLVERS_CSV | sed -r ':r; s/("[^",]+),([^",]*)/\1 \2/g; tr; s/"//g' | cut -d, -f11)
  echo "Resolver=$RESOLVER IP=${RESOLVER_IP%:*}"
done

That should get you further along your quest. Note ipv6 resolvers would have the ipv6 address in third brackets

 

[Denna]

@redhat27,

Thanks for the detailed help. Your input is appreciated.

I'm deciphering how this syntax works.​

 

[redhat27]

RESOLVER=$(sed -n "s/\"$//;/^ARGS/s/^.*-R //p" $script) # pick next word after "-R"

the s/\"$// removes trailing double quote, the /^ARGS/s/^.*-R //p will remove anything upto -R and the space following it on the line starting with ARGS. sed -n would only print what is remaining, which is the resolver name

RESOLVER_IP=$(sed -n "/^$RESOLVER,/p" $RESOLVERS_CSV | sed -r ':r; s/("[^",]+),([^",]*)/\1 \2/g; tr; s/"//g' | cut -d, -f11)

the first sed -n behaves like grep (could have used grep -E as well) it will print only the line on the csv that starts with the resolver name and a comma
the second part (sed -r and cut) is what you've already used before (so no explanation necessary)

echo "Resolver=$RESOLVER IP=${RESOLVER_IP%:*}"

The ${RESOLVER_IP%:*} bit is to show the ip only (some IPs have a port mentioned along with it after a colon, so dropping that in the display)

 

[Denna]

@redhat27,

It's taking a while to integrate this into the firewall script, but I did want to say thank you for the detailed help and explanations.

I'll post the firewall script when it's done so you'll be able to see your handiwork in action !​

 

[Denna]

The scripting above provides the method to extract the IP address for the S09dnscrypt-proxy file.

How would you iterate through other "S09dnscrypt-proxy" files with a number appended to the end of the filename ?

For example, S09dnscrypt-proxy1 ?

 

[redhat27]

Try copy pasting the script as is. It should go through all the S09dnscrypt-proxy* files

 

[Denna]

@redhat27,

Didn't notice the asterisk at the end of the filename.

Thank you.​

 

[Denna]

@redhat27,

To get the IP address added to the IP set, would the echo line be replaced with ... ?

ipset add DNSCRYPT_IPS ${RESOLVER_IP%:*}

 

[redhat27]

@redhat27,

To get the IP address added to the IP set, would the echo line be replaced with ... ?

ipset add DNSCRYPT_IPS ${RESOLVER_IP%:*}

Yes

 

[Denna]

@redhat27,

It works !

Thank you.​