Apparently the box is querying flagged DNS domains by itself

[develox]

Hi,

so, now that I've put in place a dns queries tracking system via Diversion/uiDivStats that tracks what domains each LAN clients queries for, I have made a step forward in investigating the following issue.

As a recap, I have Suricata installed on my RT-AC5300 via opkg/entware. I'm signalled alerts about suspect DNS queries of 2 types:

1. Level: Warning, "ET DNS Query to a *.top domain - Likely Hostile" (e.g. dgafgadsgkjg.top) or "ET DNS Query to a *.pw domain - Likely Hostile" (e.g. us.bookofstorage.pw)
2. Level: Alert, "ET POLICY Android Adups Firmware DNS Query 2", one example is: bigdata.adsunflower.com (explanation at: www.kryptowire.com/adups_security_analysis.html)

Setting still aside whether they're really relevant or not, Diversion now confirms what Suricata already showed, i.e. no client in the LAN is requesting these domains. Log entries for such queries are tracked with source IP being the WAN address of the Asus and dest IP my external firewall (a ZyXEL USG).

So the suspect is now confirmed to be the router itself, and the question becomes: why does he need such domains ? I would bet that neither stock firmware by itself, nor a clean Merlin install would need them, so what can it be ? Something I installed via Entware ? And how can I investigate it ?

It's not something that you'd find via an extensive (actually not viable) tcpdump on the machine as it wouldn't show the process initiating the request. The only command I can think of in linux that might help is an lsof (with -i option), but that's helpful only in the very same instant when the communication occurs, so you're highly unlikely to catch it.

Note that I'm running the latest 386.1_2 fw. The router runs Suricata, Skynet and Diversion (with pixel-srv enabled).

Thanks
Peppe

 

[dave14305]

So the suspect is now confirmed to be the router itself, and the question becomes: why does he need such domains ? I would bet that neither stock firmware by itself, nor a clean Merlin install would need them, so what can it be ? Something I installed via Entware ? And how can I investigate it ?

I’m not sure you’ve really confirmed anything yet. Is DNS Filter enabled in Router mode to force all clients through dnsmasq? Have you enabled Tools / Other Settings page “Wan: Use local caching DNS server as system resolver (default: No)” to have the router also use dnsmasq?

If you do both those things you should then be able to confirm the source of the queries.

 

[develox]

Hi Dave, thanks for the feedback.
DNS Filter is disabled right to have everything going to dnsmasq (I used to have it enabled and I disabled it last w-e just to force this behaviour and have the logs from the dnsmasq via Diversion/uiDivStats).
"Wan: Use local caching DNS server as system resolver" is left at its default (No).

In my LAN --> DHCP configuration I've blank custom DNS entries but I enabled "Advertise router's IP in addition to user-specified DNS", and indeed nslookup requests from clients get answers by the router's IP.

So I think the suspect is confirmed because dnsmasq is actually logging what seem to be all of the LAN clients' requests, and Suricata is doing the same + the WAN side of the router requests (which I don't see via dnsmasq).

Any idea on how to move some step forward from here ?

 

[dave14305]

Hi Dave, thanks for the feedback.
DNS Filter is disabled right to have everything going to dnsmasq (I used to have it enabled and I disabled it last w-e just to force this behaviour and have the logs from the dnsmasq via Diversion/uiDivStats).
"Wan: Use local caching DNS server as system resolver" is left at its default (No).

In my LAN --> DHCP configuration I've blank custom DNS entries but I enabled "Advertise router's IP in addition to user-specified DNS", and indeed nslookup requests from clients get answers by the router's IP.

So I think the suspect is confirmed because dnsmasq is actually logging what seem to be all of the LAN clients' requests, and Suricata is doing the same + the WAN side of the router requests (which I don't see via dnsmasq).

Any idea on how to move some step forward from here ?

Without DNS Filter enabled in Router mode, some clients may be using hard-coded DNS entries that won't respect the DHCP DNS server. So you may be missing those queries in your dnsmasq logs.

If you enable the local caching DNS server option, then the router's own queries will also appear in your dnsmasq logs.

Doing both of these things would give you your smoking gun of where to look next. But I don't really believe the router itself will be initiating queries to unexpected domains like this.

 

[develox]

Ok, I've just done both. Now wait for the next hit and see if any other clue pops up.
Thanks.

 

[develox]

oh, yes, and there it is, while we were all sleeping:

dnsmasq.log1:Mar  2 02:26:45 dnsmasq[7049]: query[AAAA] bigdata.adfuture.cn from 127.0.0.1
dnsmasq.log1:Mar  2 02:26:45 dnsmasq[7049]: forwarded bigdata.adfuture.cn to 1.1.1.1
dnsmasq.log1:Mar  2 02:26:46 dnsmasq[7049]: query[A] bigdata.adfuture.cn from 127.0.0.1
dnsmasq.log1:Mar  2 02:26:46 dnsmasq[7049]: /opt/share/diversion/list/blockinglist bigdata.adfuture.cn is 192.168.1.2

or

dnsmasq.log1:Mar  2 02:26:45 dnsmasq[7049]: query[AAAA] bigdata.adsunflower.com from 127.0.0.1
dnsmasq.log1:Mar  2 02:26:45 dnsmasq[7049]: forwarded bigdata.adsunflower.com to 1.1.1.1
dnsmasq.log1:Mar  2 02:26:46 dnsmasq[7049]: query[A] bigdata.adsunflower.com from 127.0.0.1
dnsmasq.log1:Mar  2 02:26:46 dnsmasq[7049]: /opt/share/diversion/list/blockinglist bigdata.adsunflower.com is 192.168.1.2

so why would 127.0.0.1 within the router need those domains ?

 

[ColinTaylor]

Do you have any Chinese mobile phones on your network?

 

[develox]

Hi Colin,

of course we do ... But at 2:26am we were all sleeping so all of them are always either turned off or disconnected.

I originally had a chinese kody box, and that was when I first got aware of these ADUPS domains and their risk. But that was years ago, and requests were clearly logged originating from that box.

Now it doesn't seem the same, and the request source is 127.0.0.1 in the router.

 

[ColinTaylor]

It's just that googling that URL brings up endless reports of it being used by Chinese phones to check for their updates, which might explain why it happens at night.

 

[develox]

It's just that googling that URL brings up endless reports of it being used by Chinese phones to check for their updates, which might explain why it happens at night.

You're right, that's their stated purpose. But it's the "how they do it" (and what also they do along with it) what has raised concerns. (And so I don't understand why 127.0.0.1 would need them).

 

[dave14305]

What runs on your router? Maybe the output of these commands would help identify any potential sources?

ps w
opkg list-installed

It's very odd to me that the query originates from the router and not a client. I'm trying to imagine if any DNSFilter or VPN configuration would make a query look like it's coming from localhost.

 

[develox]

That's a good point Dave (to look at processes and installed packages),
I've tried looking at it but to my eyes I can't see anything which is a candidate for the issue. Anyway, here it is. I've listed installed packages in a "raw view" seen by opkg itself, and the organised view as seen by amtm.
There's also a current process list.

And I agree, it's odd to me too why the router should originate these queries.

 

[develox]

Could it be that's one of the security/protection services in the router itself that's trying to translate those domains to keep an updated list of their related IPs ? Perhaps even an old manual entry like in Skynet ?
How could I check ? I've been looking a way to list manual bans on Skynet but couldn't find it so far.

 

[dave14305]

It could be Skynet if you have ever manually banned those domains (try firewall stats search manualbans) or if you have AiProtection enabled on the router and Skynet is set to Import AiProtect Data.

 

[develox]

It could be Skynet if you have ever manually banned those domains (try firewall stats search manualbans) or if you have AiProtection enabled on the router and Skynet is set to Import AiProtect Data.

I did this and didn't show anything. Does it search for configured manual bans, or logged entries for matches against manual bans ?

 

[dave14305]

I did this and didn't show anything. Does it search for configured manual bans, or logged entries for matches against manual bans ?

It searches its own events.log, which seems to be where it remembers the manual bans. For Skynet to even be relevant, does it have cron jobs scheduled at 2:26 AM ? Check with cru l

 

[develox]

As it searches its own event log it's perhaps explainable that it can't find anything because there's actually no traffic against those domains (they're even blocked in the outer firewall). But it might be enough to try to translate them to trigger the Suricata rules.
Crontab entries for Skynet and Diversion are not close enough to 2:26am, that's a good observation too. (And there's nothing in crontab more than Skynet and Diversion, except restarting Suricata e rolling its log files, but that happens at 4am).
So back to the question: how can I visualise my current manual bans configured entries (IP, domains, etc) in Skynet (it it's relevant, meaning it can fires his own domains translation even outside of its configured cron jobs) ?
Apart from this, I've gone through the Asuswrt GUI and those domains are not listed anywhere (is there a way to search a textual config to make sure ? They're not in nvram too though).

 

[dave14305]

So back to the question: how can I visualise my current manual bans configured entries (IP, domains, etc) in Skynet

cd /tmp/mnt/<USB>/skynet
grep "Manual Bans" events.log
grep ManualBanD skynet.ipset

 

[develox]

Done, empty results. As I was there, I added a:

grep -R -i bigdata *

which was empty too, but the following did hit a result:

grep -R -i "bookofstorage" * | head
skynet.ipset:add Skynet-Blacklist 69.10.62.204 comment "BanAiProtect: us.bookofstorage.pw"

which gives perhaps only half an explanation (though the cron observation still holds), but that was the less "interesting" domain of the two I mentioned originally. ADUPS ones raise more concerns.

 

[develox]

dnsmasq.log1:Mar 2 02:26:46 dnsmasq[7049]: /opt/share/diversion/list/blockinglist bigdata.adsunflower.com is 192.168.1.2

Meanwhile, I've reviewed what we've written here and what we've tried.
I noticed that one of these ADUPS domains, as I reported here, it's actually in the blocking list of diversion. Which, I thought, might come from Skynet ?
I've run 3 times in a row Skynet's malware blocklist update, and ... I got the whole set of flagged DNS hits all of the times, from 127.0.0.1 in dnsmasq's log and from the router's WAN IP in Suricata (i.e. the original question).
This, though only empirically, would suggests that's actually Skynet that probably do it while updating the blocking/malware lists.
I would still love to track DNS requests per PID, to get proof that its them and, especially, only them, if only for learning purposes. But we've gained another bit of explanation anyway and clears much of the clouds.
Meanwhile, thanks very much for the help so far.