Web History Reporting and Management (Traffic Analyzer/Aiprotection Monitor)

[ColinTaylor]

No luck yet...

Looks like you have created the file in DOS/Windows format instead of Unix format. Run the following command to correct the file (and any other files you have created this way).

dos2unix /jffs/scripts/webhistory_report.sh

 

[TicoMan]

Looks like you have created the file in DOS/Windows format instead of Unix format. Run the following command to correct the file (and any other files you have created this way).

dos2unix /jffs/scripts/webhistory_report.sh

That was it! thanks a lot for the help!

 

[Reef2009]

Is this tool still working on latest releases? 386.x etc?

 

[killeriq]

just tried today and it works...
But i dont see any IP / Hostname, do i need to somehow specify it in the command?

@Martineau any clue?

Thanks

 

[Martineau]

just tried today and it works...

@Martineau any clue?


View attachment 38205

I suspect if you are now using v386.4 it may be because of this.....

so the lookup of the hostname is presumably failing due the new format.

I'll take a look.

 

[killeriq]

i was on v386.3.2
i tired afterwards 386.4 and look like this:

Anyway thanks for fast action !

btw is there possible to somehow track full URLs like in Web browser History? instead of "main URL" only?

Thanks

 

[Martineau]

i was on v386.3.2
i tired afterwards 386.4 and look like this:

View attachment 38207

Ahh, OK......Two years untouched and the format has changed twice?

I have uploaded Web History Report v1.15 but this is now ONLY for v386.4+

 

[killeriq]

All is good now great support!
--

btw is there possible to somehow track full URLs like in Web browser History? instead of "main URL" only?

Thanks

 

[Martineau]

btw is there possible to somehow track full URLs like in Web browser History? instead of "main URL" only?

You would have to ask ASUS to implement this feature request.

 

[killeriq]

Hi @Martineau

1. after some time i ran it again and in hostname is showing just "?" even before it was showing host name properly. Any clue what might be wrong this time (Current Version : 386.4) ?
2. When i ran "sh WebHistory_Report.sh nofilter" or "nofilter report=WebReport.csv" its listing it, but is kind of slow , like one entry per 1 second.
So if i want to run backup for month, it takes ages.

Speed is dependable on Router? as i dont see any load on 4 core more then 5%. Any way to make it faster or issue is somewhere on my side?

Thanks

 

[Martineau]

Hi @Martineau

1. after some time i ran it again and in hostname is showing just "?" even before it was showing host name properly. Any clue what might be wrong this time (Current Version : 386.4) ?

So when it was working, what has changed since?

2. When i ran "sh WebHistory_Report.sh nofilter" or "nofilter report=WebReport.csv" its listing it, but is kind of slow , like one entry per 1 second.
So if i want to run backup for month, it takes ages.

Speed is dependable on Router? as i dont see any load on 4 core more then 5%. Any way to make it faster or issue is somewhere on my side?

Thanks


View attachment 39006

It's no doubt probably my manky coding skills, but it definitely depends on the size of the database.

i.e. The fancy graphics/colourful formatting will take a toll, and as advised (in post #1- reproduced) below, you may need to simply eschew them and supply your own SQL Query direct, or in combination with external tools/filters:

 

[killeriq]

i haven't changed anything, as there was no new FW since , so no clue what is wrong.
Tried also Router restart and same behavior "?"

tried as you suggested: "./WebHistory_Report.sh nofilter noscript"

Or also "./WebHistory_Report.sh nofilter report=WebReport.csv nodisplay"

it went really fast with "nodisplay" or "noscript" option, but is ignoring "Local Host name"

"2022-01-15 20:15:27,30:A7:A2:AE:A11,www.xxx.xxx"

Thanks

 

[Martineau]

i haven't changed anything, as there was no new FW since , so no clue what is wrong.
Tried also Router restart and same behavior "?"

tried as you suggested: "./WebHistory_Report.sh nofilter noscript"

Or also "./WebHistory_Report.sh nofilter report=WebReport.csv nodisplay"

it went really fast with "nodisplay" or "noscript" option, but is ignoring "Local Host name"

"2022-01-15 20:15:27,30:A7:A2:AE:A11,www.xxx.xxx"

Thanks

Do any of the following produce output

cat /etc/dnsmasq.conf | grep "^dhcp-host"

nvram get dhcp_staticlist

grep "30:A7:A2:AE:A1:D1" /etc/dnsmasq.conf

 

[killeriq]

i tried "cat /etc/dnsmasq.conf | grep "^dhcp-host" and give this...
this - nvram get dhcp_staticlist - make the same but all in single like, no host names

grep "30:A7:A2:AE:XX:XX" /etc/dnsmasq.conf

 

[killeriq]

One more thing all values were like 15-18h old last 2 days, so i thought maybe is full log or what

So I tried to "clear" in UI , then shutdown, 2min wait and boot ...

Now the hostname still shows "?" on all values...

dont want to do factory reset for now...aaah

 

[killeriq]

so i ended up with Factory reset, as many things were wrong, no clue why...
will see how it goes few days, then i try your script again as i have no clue what is causing those strange problems

Flash issue? - Config not saving, VPN Director, hostnames

Hi , i have issue that when is clean up list of old devices from VPN Director and hit apply, all seems fine Router: RT-AX88U Firmware Version:386.4 Soon as i reboot, i have the old list there again :( Anything else what could be done except "factory reset" and let setup all again? Thanks

www.snbforums.com

 

[killeriq]

today i tried your script again...all seems fine in the command window. All is there
YYYY/MM/DD HH:MM:SS MAC address Host Name IP address URL

But when i want to export "nofilter report=WebReport.csv nodisplay"
It creates CSV file only with Date,MAC,URL

How to have it with Host Name and IP as well? Its easier to identify what is what

Thanks!

 

[Martineau]

today i tried your script again...all seems fine in the command window. All is there
YYYY/MM/DD HH:MM:SS MAC address Host Name IP address URL

But when i want to export "nofilter report=WebReport.csv nodisplay"
It creates CSV file only with Date,MAC,URL

How to have it with Host Name and IP as well? Its easier to identify what is what

Thanks!

Unfortunately the ASUS database doesn't contain the Hostname/IP

i.e. if you issue

sqlite3 /jffs/.sys/WebHistory/WebHistory.db "select datetime(timestamp, 'unixepoch', 'localtime') AS time, mac, url FROM History;"

the displayed records only contain three fields/columns - Date,MAC,URL

However, you can try these three commands

sqlite3 /jffs/.sys/WebHistory/Addons.db "CREATE TABLE IF NOT EXISTS hosts(mac TEXT,hostname TEXT,ip TEXT);"

sqlite3 /jffs/.sys/WebHistory/Addons.db "DELETE FROM hosts";arp -a | tr -d '()' | awk '{print toupper($4)","$2","$1}' | sqlite3 -csv /jffs/.sys/WebHistory/Addons.db ".import '|cat -' hosts"; sqlite3 /jffs/.sys/WebHistory/Addons.db "SELECT * FROM hosts"

sqlite3 /jffs/.sys/WebHistory/WebHistory.db "attach '/jffs/.sys/WebHistory/WebHistory.db' as asus;attach '/jffs/.sys/WebHistory/Addons.db' as killeriq;select datetime(timestamp, 'unixepoch', 'localtime') AS time, x.hostname, x.ip, h.mac, h.url from asus.history h join killeriq.hosts x on h.mac = x.mac;" | sort

 

[garett_09]

Hey, @Martineau, I came across your script in GiHub and tried using your TrafficAnalyzer script. This is what it returned to me after using it same error for WebHistory but this time for traffic analyzer. I know it's been a long time since you last edited but I hope you can fix it. Thank you!

 

[Martineau]

Hey, @Martineau, I came across your script in GiHub and tried using your TrafficAnalyzer script. This is what it returned to me after using it same error for WebHistory but this time for traffic analyzer. I know it's been a long time since you last edited but I hope you can fix it. Thank you!

View attachment 51351

It has been a while since I last reviewed these scripts.

I have uploaded new versions of both the TrafficAnalyzer and WebHistory scripts

e.g. Web History Report

so they may be fixed?

P.S. Which router and firmware are you using?