Monitoring Web Usage with Asuswrt-Merlin

[john9527]

Good that we have the same configuation for eth0
And for you does webmon tracks any kind of web acces to both "LANs"(eth0 and eth0:0)?

I'll have to fire it up to look, never checked that before.

I see your modem is is the 192.168.1.x subnet.....what is your router address and netmask?

 

[ocram59]

I'll have to fire it up to look, never checked that before.

I see your modem is is the 192.168.1.x subnet.....what is your router address and netmask?

Router address is 192.168.50.1 netmask 255.255.255.0

 

[ocram59]

BINGOOOOO ... I GOT IT WORKING

I noticed that my eth0 IP addres was not my WAN adddres, maybe because i use PPP on ADSL? (I wish i had fiber....)

Being my WAN ip on ppo0 i tried this:

iptables -t filter -I FORWARD -o ppp0 -j monitor

And now webmon works!!!!

admin@RT-N66U:/tmp/home/root# cat /proc/webmon_recent_domains
1427074661      192.168.50.2    heartbeat.dm.origin.com
1427074644      192.168.50.2    www.snbforums.com
1427074622      192.168.50.2    forum.ea.com
1427074574      192.168.50.2    192.168.1.1

So basically the result is that we must enable webmon filter on the interface that actually handles the IP traffic (in my case ppp0) ? (maybe that if i was not using ppp, eth0 woud be fine..)

 

[john9527]

BINGO! YAHOO! Sorry for not asking the right questions You end up thinking in terms of your own configuration.

And BTW...it doesn't track the modem accesses.....

 

[ocram59]

BINGO! YAHOO! Sorry for not asking the right questions You end up thinking in terms of your own configuration.

And BTW...it doesn't track the modem accesses.....

John thanks alot for all your support and looool so now i am ahead because here it tracks both modem and "outside"

This happy ending makes me happy but it also makes me thinnk how much behind we are here .. still using ADSL and ppp .... all other people did not fall inthis problem , probably because they have a true "eth" connection to the net

Thanks alot again , maybe you could add a note in your txt in case someone else falls in the same configuration

Best Regards
ocraM

 

[john9527]

maybe you could add a note in your txt in case someone else falls in the same configuration

Absolutely....will do (and give you the credit!)

EDIT: Using-Webmon.txt updated in the 374 Merlin Fork download area.

 

[tijaune]

Hi john9527,

After 2 weeks where adblocks now earned a permanent spot in my jffs space, I revisit this webmon thread since it was one interesting feature I miss from Shibby Tomato. I hope you can confirm my understanding of your recipe as per follow:

1) iptables rules in firewall-start write to a virtual memory file at /proc/webmon_recent_domains.
2) showwebmon.sh script sort this file to /tmp and echo every line with proper datetime formatting.

Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/root                   27        27         0 100% /
tmpfs                      117         4       112   4% /tmp
devfs                      117         0       117   0% /dev
/dev/mtdblock5              32         1        31   4% /jffs
/dev/sda1                 1885         4      1786   0% /tmp/mnt/ext2

my /tmp has 112 MB available, about half of the router physical ram. So I guess everything here is done in ram? I don't have to modify the script to use /tmp/mnt/ext2 which is my 2 GB usb flash?

Thank you for your works,

 

[Curado Kev]

Router/Modem enthusiast with a RT68 and been reading this forum for 1+ years, however I would like to track my kids on sites they access. Already have DNS filtering on but this is a little more technical for me since I'm not a IT network engineer. Does anyone have step by step instructions and/or are there any plans to write this into firmware whee you can "turn it on' and then put in "gmail address" for delivery ? I'm willing to do the work but I'm not a script guy. Help......

 

[Cake]

This is what I was looking for yesterday. It worked right away follow the instructions on the 2nd page of this thread.
Thanks x 100. Forgot I bookmarked this thread a while back.
Here is my firewall-start in /jffs/scripts/

#!/bin/sh

# Enable webmon monitoring
  if [ -z "$(iptables -n -L FORWARD | grep 'web_monitor')" ]; then
     modprobe ipt_webmon
     iptables -t filter -N web_monitor
     iptables -t filter -I FORWARD -o eth0 -j web_monitor
     iptables -I FORWARD -o tun11 -j web_monitor
     iptables -I FORWARD -o tun12 -j web_monitor
     iptables -A web_monitor -p tcp -m webmon --max_domains 2000 --max_searches 2000
     iptables -I web_monitor -m state --state NEW -j LOG --log-level 6 --log-prefix "webmon"
fi

I am going to remove

iptables -I web_monitor -m state --state NEW -j LOG --log-level 6 --log-prefix "webmon"

because it only shows ip address and not domains in the log.

This is the same as web history in the router GUI, except for that the TrendMicro dpi or whatever, associates the host name (or the name you assigned the device) with the domain name.
Yesterday I went with logging dnsmasq. When comparing webmon and dnsmasq logging, the latter shows me allot of other stuff including sip phones, it maybe more helpful if your trying to find a problem device. Also the latter can send device ip the and domain to the syslog, however it fills up fast.

I hope a somebody with skills posts a better way to move the output of John's script to a web page on a thumb drive, or something. Feature request RMerlin lol

 

[john9527]

I hope a somebody with skills posts a better way to move the output of John's script to a web page on a thumb drive, or something

Here's a script a user developed to move the data to a usb drive and reset the log on the router. He ran it daily via cru.

#!/bin/sh
domains="/proc/webmon_recent_domains"
dsort="/tmp/dsort.tmp"
cat $domains | sort > $dsort
 
while read webdate webip hostname
do
  # Reformat the webmon output to convert epoch date to general date string
  webdate=$(date -d @$webdate)
  #webdate=${webdate//GMT $(date +"%Y")/} # exclude year in output
  webdate=${webdate//GMT /}  # include year in output
  echo -e "$webdate \t$webip \t$hostname" >> /tmp/mnt/sda1/logs/domain.log
done < $dsort
rm $dsort
 
# Clear webmon filter
iptables -t filter -I web_monitor -p tcp -m webmon --clear_search --clear_domain
sleep 3
service restart_firewall

 

[ColinTaylor]

EDIT: Using-Webmon.txt updated in the 374 Merlin Fork download area.

I've noticed that this text file hasn't been included in the download area since release 12.

PS Personally I prefer this single line for $webdate (it avoids the GMT/DST/ issue)

webdate=$(date +"%Y%m%d %H:%M:%S" -d @$webdate)

 

[kvic]

Yesterday I went with logging dnsmasq. When comparing webmon and dnsmasq logging, the latter shows me allot of other stuff including sip phones, it maybe more helpful if your trying to find a problem device. Also the latter can send device ip the and domain to the syslog, however it fills up fast.

dnsmasq can log to syslog or to file. To log to file, add the below line to /jffs/configs/dnsmasq.conf.add

• log-facility=/opt/var/log/dnsmasq.log

change the file to any full path corresponding to your pen drive.

I found the idea in this thread interesting. If I had seen it earlier, I would have given it a try. However, as you rightly point out dnsmansq logging gives a superset of info. Better yet it doesn't slow down iptable processing.

Such logging is not something I would look at on a daily basis. The data is archived for CSI purpose in situations of mishap.

 

[ColinTaylor]

I've been playing with this for a few hours now and it's quite fun. Unfortunately, as it's only possible to log URLs in HTTP requests, not HTTPS, its use is fairly limited.

So if you're looking to [spy on|audit] what sites your clients are going to you'll have to use a different method.

 

[kvic]

So if you're looking to [spy on|audit] what sites your clients are going to you'll have to use a different method.

If parents want to audit web access by kids (*cough*), could try a transparent proxy such as Polipo available from Entware. It's light weight and under active maintenance.

 

[jasonchewy]

@RMerlin Request: any chance of adding this to the build/gui?

I'm testing some ip cameras to see if they're secretly talking to the outside, for example if it's using some cloud service perhaps. (yes I know I could just run parental block on the device to ensure it's not talking, but it'll be nice to know if a device is behaving nicely or not to start off)
Currently I have to run a secondary router running tomato under my network to test them. Using a combination of the web usage and syslog. Having this added officially would be nice.
Thanks.

 

[ColinTaylor]

@RMerlin Request: any chance of adding this to the build/gui?

I'm testing some ip cameras to see if they're secretly talking to the outside, for example if it's using some cloud service perhaps. (yes I know I could just run parental block on the device to ensure it's not talking, but it'll be nice to know if a device is behaving nicely or not to start off)
Currently I have to run a secondary router running tomato under my network to test them. Using a combination of the web usage and syslog. Having this added officially would be nice.
Thanks.

Isn't this built into the firmware now? Traffic Analyzer - Statistic

(I don't know for sure because I don't use Merlin's firmware)

 

[jasonchewy]

Isn't this built into the firmware now? Traffic Analyzer - Statistic

(I don't know for sure because I don't use Merlin's firmware)

Oh, in that case I'm going to have to update my firmware, I'm still running 380.57, so not sure if what you're referring to is something new or the same "Traffic Monitor" that I have, which only gives me how much data has gone through the router, not where the devices are going to.
Maybe this is a good time to force myself to finally setup the newer router that has been sitting around for months....

 

[john9527]

Isn't this built into the firmware now? Traffic Analyzer - Statistic

It's part of TrendMicro, so only available on the ARM based routers (not N66 or AC66)

 

[Derek Kerton]

It's part of TrendMicro, so only available on the ARM based routers (not N66 or AC66)

On my RT-AC88U, the Web History log is stored for each client, and located under
Adaptive QoS > Web History

Partial of mine has better spacing, and contains:
Access Time MAC Address / Client's Name Domain Name
2017-9-20 19:00:36 Chromecast-Ultra api-global.netflix.com
2017-9-20 18:59:30 Chromecast-Ultra api-global.netflix.com
2017-9-20 18:59:16 Chromecast-Ultra lh4.googleusercontent.com
2017-9-20 18:58:24 Chromecast-Ultra api-global.netflix.com
2017-9-20 18:57:18 Chromecast-Ultra api-global.netflix.com
2017-9-20 18:56:44 Chromecast-Ultra clients3.google.com

 

[jasonchewy]

On my RT-AC88U, the Web History log is stored for each client, and located under
Adaptive QoS > Web History

Partial of mine has better spacing, and contains:
Access Time MAC Address / Client's Name Domain Name
2017-9-20 19:00:36 Chromecast-Ultra api-global.netflix.com
2017-9-20 18:59:30 Chromecast-Ultra api-global.netflix.com
2017-9-20 18:59:16 Chromecast-Ultra lh4.googleusercontent.com
2017-9-20 18:58:24 Chromecast-Ultra api-global.netflix.com
2017-9-20 18:57:18 Chromecast-Ultra api-global.netflix.com
2017-9-20 18:56:44 Chromecast-Ultra clients3.google.com

That's perfect.
I'm going to get my AC68U setup this weekend then. (don't look forward to moving all my DHCP mapping list and port forwarding list over)