The beauty of a successful search. I was hoping this data was part of the per-IP monitoring, but alas, no joy. At least I found the definitive answer.
Monitoring Web Usage with Asuswrt-Merlin
- Thread starter bhall7
- Start date
The beauty of a successful search. I was hoping this data was part of the per-IP monitoring, but alas, no joy. At least I found the definitive answer.
I think this function is really cool. It helps to monitoring which URLs kids are going to. Is there really no way that this feature can be enabled without Merlin's help? This has been on my wish list for a very long time and am just doing research on it now. I don't want to switch to Shibby's firmware just for this feature :-(
homer2320776
Occasional Visitor
I know the feeling. I've been trying to find / make something similar to the QOS transfer rates that was in Tomato
Been playing with Entware and trying darkstat, iptraf, and iftop but they are not as robust as what I was looking for.
Installed Debian and tried ntop, then compiled ntop-ng and they are just too bulky.
Hopefully one day I can find something or Merlin will add it in.
As for what you are looking for, you might be able to install Entware and play around with Zabbix. I know it can do web monitoring.
john9527
Part of the Furniture
This thread was started a long time ago....and I think things have changed. It looks like Merlin added support for the libipt_webmon binary at some point (but only for ARM). I backported it to my fork and it appears to be working.
add the following to /jffs/scripts/firewall-start (if you don't have an existing firewall-start, add the shebang as the first line)
Copy this little script to format the output as showwebmon.sh, and make it executable
And, some sample output
add the following to /jffs/scripts/firewall-start (if you don't have an existing firewall-start, add the shebang as the first line)
Code:
# Enable webmon monitoring
if [ -z "$(iptables -n -L FORWARD | grep 'web_monitor')" ]; then
modprobe ipt_webmon
iptables -t filter -N web_monitor
iptables -t filter -I FORWARD -o eth0 -j web_monitor
iptables -A web_monitor -p tcp -m webmon --max_domains 2000 --max_searches 2000
fiCopy this little script to format the output as showwebmon.sh, and make it executable
Code:
#!/bin/sh
# Reformat the webmon output to convert epoch date to general date string
domains="/proc/webmon_recent_domains"
dsort="/tmp/dsort.tmp"
cat $domains | sort > $dsort
while read webdate webip hostname
do
webdate=$(date -d @$webdate)
#webdate=${webdate//GMT $(date +"%Y")/} # exclude year in output
webdate=${webdate//GMT /} # include year in output
echo -e "$webdate \t$webip \t$hostname"
done < $dsort
rm $dsort
exit 0
Code:
admin@AC68R-00000:/jffs/scripts# showwebmon.sh
Tue Mar 3 10:54:11 2015 192.168.1.100 cdn.tt.omtrdc.net
Tue Mar 3 10:54:11 2015 192.168.1.100 fonts.googleapis.com
Tue Mar 3 10:54:11 2015 192.168.1.100 foxnews.tt.omtrdc.net
Tue Mar 3 10:54:12 2015 192.168.1.100 secure-us.imrworldwide.com
Tue Mar 3 10:54:13 2015 192.168.1.100 a.visualrevenue.com
Tue Mar 3 10:54:13 2015 192.168.1.100 global.fncstatic.com
Tue Mar 3 10:54:13 2015 192.168.1.100 p.visualrevenue.com
Tue Mar 3 10:54:13 2015 192.168.1.100 srv-2015-03-03-17.config.parsely.com
Tue Mar 3 10:54:14 2015 192.168.1.100 foxnewsplayer-a.akamaihd.net
Tue Mar 3 10:54:14 2015 192.168.1.100 srv-2015-03-03-17.pixel.parsely.com
Tue Mar 3 10:54:14 2015 192.168.1.100 static.parsely.com
Tue Mar 3 10:54:14 2015 192.168.1.100 t1.visualrevenue.com
Tue Mar 3 10:55:56 2015 192.168.1.100 b.scorecardresearch.com
Tue Mar 3 10:55:56 2015 192.168.1.100 cdn.sstatic.net
Tue Mar 3 10:55:56 2015 192.168.1.100 pixel.quantserve.com
Tue Mar 3 10:56:11 2015 192.168.1.100 superuser.com
Tue Mar 3 10:56:40 2015 192.168.1.100 www.ipchicken.com
Tue Mar 3 10:57:12 2015 192.168.1.100 rm.recs.richrelevance.com
Tue Mar 3 10:57:16 2015 192.168.1.100 images.highspeedbackbone.net
Tue Mar 3 10:57:17 2015 192.168.1.100 p.tigerdirect-cmr.net
Tue Mar 3 11:01:18 2015 192.168.1.100 i1.wp.com
Tue Mar 3 11:01:34 2015 192.168.1.100 connect.facebook.net
Tue Mar 3 11:01:34 2015 192.168.1.100 www.gravatar.com
Tue Mar 3 11:01:41 2015 192.168.1.100 ipaddress.com
Tue Mar 3 11:02:21 2015 192.168.1.100 videosearch.ubuntu.com
Tue Mar 3 11:02:51 2015 192.168.1.100 www.snbforums.com
Tue Mar 3 11:06:40 2015 192.168.1.100 www.whereisip.net
Last edited:
john9527
Part of the Furniture
Just a quick note for anyone trying this......NAT Acceleration has to be disabled for it to work
Also, if you are using a VPN Client, it will work there as well if you add a rule for the tunnel (change the tunnel name as appropriate)....and you can leave NAT Acceleration enabled since the tunnel bypasses it. For example...
Also, if you are using a VPN Client, it will work there as well if you add a rule for the tunnel (change the tunnel name as appropriate)....and you can leave NAT Acceleration enabled since the tunnel bypasses it. For example...
Code:
iptables -I FORWARD -o tun11 -j web_monitormobileman88
Regular Contributor
What is the performance of the AC66 , AC68 and AC87 with NAT Acceleration disabled? I understand they will drop significantly?
john9527
Part of the Furniture
My understanding is max WAN throughput is 100-150 Mbs with NAT Acceleration disabled (towards the lower end for MIPS routers, higher end for ARM) vs 800+Mbs with.
An AC68U/R should hit close to 350 Mbps, and an AC87U probably close to 400 Mbps when NAT acceleration is disabled.
mobileman88
Regular Contributor
Thanks @RMerlin I will disable it and see how well this module works then . I have a 400Mbps connection with the AC68U and I typically don't get more than 320Mbps from my ISP. It's easily reversible if I disable QOS and webmon plugins right?
mobileman88
Regular Contributor
I just turned off my NAT acceleration and inserted your script into my /jffs/scripts/firewall-start . I found that the iptables entries have been updated however /proc/webmon_recent_domains and /proc/webmon_recent_searches seems to show up empty when I tried to cat them or use your showwebmon.sh script. How can I troubleshoot this? I monitored for over 30 minutes but see no data. The AC68 console shows that NAT acceleration have already been turned off.
john9527
Part of the Furniture
First thing I would try would be to remove the entries from firewall-start, reboot, then try to enter the commands manually via telnet/ssh to check there are no error messages (all except the 'if' statement, that's just a safeguard to prevent it running when it's already loaded).
przemekwawa
Regular Contributor
Thanks for those scripts. Is there possibility to send those logs using syslog? How to change default file to e.g. pendrive?
john9527
Part of the Furniture
Anything is possible.....it's just a SMOP (Simple Matter of Programming
)The script I posted, was really just a basic, 'get started' script. To really make it useful, you would need to:
-Periodically lock the source file, archive it, clear it then unlock it again. The data file itself will wrap in 2000 entries (with the firewall-start I made).
- When you archive it, you could send it to a pendrive if desired. Then you would process the data from there, put it in the syslog if you wanted and put in place some kind of data management to clear old files.
Maybe one of the forum bash experts can help out?
przemekwawa
Regular Contributor
It is maybe good idea to lock it, but for now, I have added rule to iptablesAnything is possible.....it's just a SMOP (Simple Matter of Programming
The script I posted, was really just a basic, 'get started' script. To really make it useful, you would need to:
-Periodically lock the source file, archive it, clear it then unlock it again. The data file itself will wrap in 2000 entries (with the firewall-start I made).
- When you archive it, you could send it to a pendrive if desired. Then you would process the data from there, put it in the syslog if you wanted and put in place some kind of data management to clear old files.
Maybe one of the forum bash experts can help out?
Code:
iptables -I web_monitor -m state --state NEW -j LOG --log-level 6 --log-prefix "webmon"There are some disadvantages: it logs using IP address, not domain name, but can be helpful for now
john9527
Part of the Furniture
Great solution.....I'm definitely not an iptables expert so never thought of that. Thanks!
Even though I don't have QoS set on my RT-AC68U, I see that inside the Adaptive QoS page is a Web History tab that seems to log a device's connections by domain name at least as long as the device is connected, I think. Could you somehow grab hold of that log? Would that fulfill your requirements any better?
(And thanks to this handy web history page, I've just noticed my malicious domain blocking ("ad blocking") through my local DNS server (Raspberry Pi) was not working. On the WAN page, I'd forgotten to set to "No" the option: Connect to DNS server automatically, when I did the major upgrade a couple of days ago. I took all the screenshots of the settings before updating, just didn't use them methodically!)
I don't know how long this web history tab has been there unnoticed by me, but, assuming RMerlin put it there, many thanks, it's a really handy feature.
Martin
(And thanks to this handy web history page, I've just noticed my malicious domain blocking ("ad blocking") through my local DNS server (Raspberry Pi) was not working. On the WAN page, I'd forgotten to set to "No" the option: Connect to DNS server automatically, when I did the major upgrade a couple of days ago. I took all the screenshots of the settings before updating, just didn't use them methodically!)
I don't know how long this web history tab has been there unnoticed by me, but, assuming RMerlin put it there, many thanks, it's a really handy feature.
Martin
Last edited:
mobileman88
Regular Contributor
Thanks John. This is helpful troubleshooting tips. Anyway I found the problem, it could be a bug with the Switch options page.
There are 3 options : Jumbo Frames, Spanning Tree and NAT Acceleration. My jumbo frames was off while the STP and NAT are on. If I simply turn NAT off, after I go back to the same page, it still shows that NAT is off. Tried it a few more times, still the same. If I turn off STP and NAT, then apply, going to that page shows all 3 are off. Then I turned STP back on. Webmon is now working. As someone has said, it'd be great for some sort of script to filter off all the unique sites and perhaps even look up against a known Malicious sites or Web Reputation database to generate the info which then gets saved as a file or emailed out. I have some ideas on this but don't have the time to refresh all my unix scripting skills. If someone is interested to do this, I am open to discuss some of my ideas on making something useful out of it.
Originally :
cat /proc/webmon_recent_domains |wc -l
749
cat /proc/webmon_recent_domains |cut -c 26- |sort |uniq |wc -l
575
The above only drills down to the uniq domains that has been monitored, these can then be piped to a reputation/malicious site checker.
I think this might be useful to those who are concerned about what sites the devices on their networks are connecting to.
For those who are using Gmail, the below script snippet can mail it out (for safe practice, setup a separate email address for sending this out instead of your regular gmail account in case your router got hacked and steals your script) :
FROM="your_gmail_address here"
AUTH="your_gmail_sender_id_here"
PASS="GMAIL_PASSWORD_HERE"
FROMNAME="Happy Router"
TO="gmail_recipient_here"
echo "Subject: SOME_SUBJECT" >/tmp/mail.txt
echo "From: "$FROMNAME"<$FROM>" >>/tmp/mail.txt
echo "Date: `date -R`" >>/tmp/mail.txt
echo "" >>/tmp/mail.txt
echo "Whatever_static_email_body_here" >>/tmp/mail.txt
echo "" >>/tmp/mail.txt
echo "My WAN IP is: `nvram get wan0_ipaddr`" >>/tmp/mail.txt
echo "Uptime is: `uptime | cut -d ',' -f1 | sed 's/^.\{12\}//g'`" >>/tmp/mail.txt
echo "" >>/tmp/mail.txt
echo "---- " >>/tmp/mail.txt
echo "Your Router." >>/tmp/mail.txt
echo "" >>/tmp/mail.txt
cat /tmp/mail.txt | sendmail -H"exec openssl s_client -quiet \
-CAfile /jffs/configs/Equifax_Secure_Certificate_Authority.pem \
-connect smtp.gmail.com:587 -tls1 -starttls smtp" \
-f"$FROM" \
-au"$AUTH" -ap"$PASS" $TO
rm /tmp/mail.txt
Last edited:
Martineau
Part of the Furniture
If creating a separate GMAIL account is not possible/desired, then if using this GMAIL script you should not use the same password as your GMAIL login, but generate a separate unique GOOGLE application password:
https://support.google.com/accounts/answer/185833?hl=en
mobileman88
Regular Contributor
great idea
Thanks a lot for posting this! I do have one issue, I can't seem to run the showwebmon.sh script as a script. If I enter the commands from showwebmon.sh one line at a time, including the while loop, I do get the same output you show in your example. However, I am not able to execute the script itself.
Here is what I get when I try to run the script:
Code:
admin01@RT-AC68U-2D18:/jffs/scripts# ls -l
-rwxr-xr-x 1 admin01 root 304 Mar 15 10:33 firewall-start
-rwxr-xr-x 1 admin01 root 448 Mar 15 11:38 showwebmon.sh
admin01@RT-AC68U-2D18:/jffs/scripts# showwebmon.sh
-sh: showwebmon.sh: not found
admin01@RT-AC68U-2D18:/jffs/scripts# /jffs/scripts/showwebmon.sh
-sh: /jffs/scripts/showwebmon.sh: not found
admin01@RT-AC68U-2D18:/jffs/scripts# sh showwebmon.sh
': No such file or directoryn_recent_domains
showwebmon.sh: line 12: syntax error: unexpected "done" (expecting "do")
admin01@RT-AC68U-2D18:/jffs/scripts#I'm not sure what the problem is because the commands do work when typed in manually. In the router's config I have jffs enabled as well as "enable jffs custom scripts and configs". I must be overlooking something obvious. Any ideas?
Similar threads
Similar threads
Similar threads
-
-
package to expand Parental Controls - Web & Apps Filters?
- Started by delikid
- Replies: 4
-
Is firmware reinstallation in recovery mode deeper than on the web interface?
- Started by lenovomen
- Replies: 20
-
-
Feature request: Two factor authentication web login. (TOTP)
- Started by DJones
- Replies: 8
-
dynamic cgnat ip is changing correctly with ddns custom script but shown ip in web interface never changes
- Started by lgkahn
- Replies: 5
-
-
-
-
Allow admin web GUI access on another interface (tailscale0)
- Started by mr8
- Replies: 0
Latest threads
-
-
AX88U will only connect at 80Mhz
- Started by antik
- Replies: 2
-
QNAP Firmware 5.2.2.xxx Update destroys Login and Data- Started by PR3MIUM
- Replies: 1
-
-
Best way to fully disable multicasting, STP, and other features?
- Started by SDF07S
- Replies: 0
Support SNBForums w/ Amazon
If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!
Sign Up For SNBForums Daily Digest
Get an update of what's new every day delivered to your mailbox. Sign up here!