logging URLs to separate server?

[RRands]

HI - I would like to log all URLs from my Asus Merlin to a logging server (or to the JFFS partition at a minimum). I did a quick search, but didn't find anything - is this a common or unusual request? Anyone know how I could get this done? Essentially, I am looking for a more permanent / longer log like what you can see in the Web UI (for Merlin) under "Adaptive QOS > Web History" (http://192.168.1.1/AdaptiveQoS_WebHistory.asp on my router - your ip may vary)

Any help or pointers are greatly appreciated!


-randy

 

[Martineau]

HI - I would like to log all URLs from my Asus Merlin to a logging server (or to the JFFS partition at a minimum). I did a quick search, but didn't find anything - is this a common or unusual request? Anyone know how I could get this done? Essentially, I am looking for a more permanent / longer log like what you can see in the Web UI (for Merlin) under "Adaptive QOS > Web History" (http://192.168.1.1/AdaptiveQoS_WebHistory.asp on my router - your ip may vary)

Any help or pointers are greatly appreciated!


-randy

You could create a cron schedule that issues the basic record extraction

sqlite3 /jffs/.sys/WebHistory/WebHistory.db "select datetime(timestamp, 'unixepoch', 'localtime') AS time, mac, url FROM History;"

or enhance the information:

Web History Reporting and Management (Traffic Analyzer/Aiprotection Monitor)

No luck yet... Looks like you have created the file in DOS/Windows format instead of Unix format. Run the following command to correct the file (and any other files you have created this way). dos2unix /jffs/scripts/webhistory_report.sh

www.snbforums.com

and pipe the output direct to a flash drive attached to the router, or create a wrapper script to exploit rsync etc.

Whilst cron is old-skool, you may be able to extract the new records as they appear if it is deemed critical.

 

[RRands]

thanks, @Martineau - QQ for you - wouldn't this possibly lose some of the data (depending on the frequency that I run it), or create multiple entries of the same URL if I run it too frequently?

 

[Martineau]

thanks, @Martineau - QQ for you - wouldn't this possibly lose some of the data (depending on the frequency that I run it), or create multiple entries of the same URL if I run it too frequently?

Indeed either is possible.

I wrote a script

Web History Reporting and Management (Traffic Analyzer/Aiprotection Monitor)

Recently there was a query regarding the Web History tracking to which I replied, so I thought I would revisit my crude script I posted. I believe the Web History is now more mature, and probably can provide useful info, but extracting the info into a printable report is sadly lacking. So here...

www.snbforums.com

which reports on the current contents of the History SQL database, so clearly if the database has been trimmed of old records between invocations of the script then data could be deemed lost.

However, if the data is exported, then it would be prudent to merge the exported records with the previously exported data or alternatively, keep track of the time of the last export, and alter the SQL extraction statement to only retrieve records that have been inserted into the SQL database since the last export.

To eliminate URL duplicates, you can pipe the sorted exported records through the uniq utility or if you export in .csv format to say an Excel spreadsheet, then it should be trivial to remove duplicates before creating/viewing the report.