uiScribe uiScribe - Custom System Log page for "scribed" logs

[elorimer]

0.4.2 worked correctly for me. Don't have the bug @L&LD noted.

If I follow, I can control which logs appear by creating a /syslog-ng2.d and putting the configuration files for those I don't want to appear there, and then including it in my syslog-ng.conf file.

 

[Jack Yaz]

Easy update, as usual.

Found a bug for you. If I click on the 'Show All' or 'Hide All' buttons, an 'x' will appear and delete the button name(s).

bug? what bug?

 

[no_name]

Sent from my iPad using Tapatalk

 

[L&LD]

In the screenshot included, I can highlight the 'Hide' or 'Show All' button and even click the 'X' beside the 'Hide' text to delete this text box.

I have already deleted the 'All' in the default 'Hide All' text.

 

[L&LD]

After rebooting both the computer and the router and also using different browsers (Edge Chromium and Firefox) and also flushing the cache on everything I could think of, the issues remained.

Before uninstalling, rebooting and then re-installing again, I decided to try the 'uf' method instead.

Fixed!

@Jack Yaz I hope you didn't pull too many hairs out trying to track this.

 

[Jack Yaz]

After rebooting both the computer and the router and also using different browsers (Edge Chromium and Firefox) and also flushing the cache on everything I could think of, the issues remained.

Before uninstalling, rebooting and then re-installing again, I decided to try the 'uf' method instead.

Fixed!

@Jack Yaz I hope you didn't pull too many hairs out trying to track this.

Nah, it was a quick fix. I neglected the type attribute on the input elements

 

[elorimer]

Ok, stupid question: How does one refresh or restart this? If I change my scribe setup, and do "scribe restart", does the webgui refresh? Or, when does it refresh?

 

[Jack Yaz]

Ok, stupid question: How does one refresh or restart this? If I change my scribe setup, and do "scribe restart", does the webgui refresh? Or, when does it refresh?

Running uiscribe to show the menu is enough. I should probably add a hook to syslog-ng's start script

 

[elorimer]

If I follow, I can control which logs appear by creating a /syslog-ng2.d and putting the configuration files for those I don't want to appear there, and then including it in my syslog-ng.conf file.

Seems I can't do this after all. Very clever to use the --preprocess-into option, but this means whatever is processed for the configuration in any included directory goes into the file you then parse for destinations. Could we only include for the webgui those that have a characteristic, like ending in ".log"? I'd like to be able to exclude some files, particularly the pixelserv log, which is not useful in the webgui. I could do that perhaps by some naming convention. Part of the utility of syslog-ng is getting stuff out of the logs to begin with.

Where this also gets a little complicated is that if you install or update syslog-ng outside of scribe you may get a new syslog-ng.conf, and you may get the scl files as well, which have lots of prepackaged things, although I don't know if any are file destinations.

Another aspect of this is that, if I follow, you are starting syslog-ng outside of the entware start script (that is, calling it directly, rather than as part of S01syslog-ng), so you will be starting it from services-start rather than from post-mount.div. Are things ordered correctly? We need the kill_logger sequence in there at the beginning.

 

[akb]

I wanted to post that this is gods work. Thank you so much for making the UI soooo useful

 

[Jack Yaz]

Seems I can't do this after all. Very clever to use the --preprocess-into option, but this means whatever is processed for the configuration in any included directory goes into the file you then parse for destinations. Could we only include for the webgui those that have a characteristic, like ending in ".log"? I'd like to be able to exclude some files, particularly the pixelserv log, which is not useful in the webgui. I could do that perhaps by some naming convention. Part of the utility of syslog-ng is getting stuff out of the logs to begin with.

Where this also gets a little complicated is that if you install or update syslog-ng outside of scribe you may get a new syslog-ng.conf, and you may get the scl files as well, which have lots of prepackaged things, although I don't know if any are file destinations.

Another aspect of this is that, if I follow, you are starting syslog-ng outside of the entware start script (that is, calling it directly, rather than as part of S01syslog-ng), so you will be starting it from services-start rather than from post-mount.div. Are things ordered correctly? We need the kill_logger sequence in there at the beginning.

That was my initial approach and I changed it to accommodate users with their own naming scheme...

I'm not starting syslog-ng at all, to my knowledge. The only syslog-ng command is the preprocess to gather the list of files. Does that also start it?

 

[elorimer]

Does that also start it?

I assumed it did, as there is nothing in the admin guide or man page that suggests this is anything other than a start option. But I just did the following:

/opt/etc/init.d/S01syslog-ng stop  <--whiich stopped syslog-ng and wrote a stop message to the log
syslog-ng --preprocess-into "some file"  <--which wrote the full config to that file
scribe status <--which showed syslog-ng as dead
scribe restart <-- which restarted syslog-ng and wrote a start message to the log

So you are right, and my assumption was wrong. I think this is a clever thing; now that scribe has a reload to hot start a config change, should it hook to reload the ui?

Also, I discovered the reason I got funny log blocks in your earlier versions is because I'm loading the full scl blocks, and a few contain destinations that were being picked up in your earlier parsing. That's gone now.

 

[Jack Yaz]

I assumed it did, as there is nothing in the admin guide or man page that suggests this is anything other than a start option. But I just did the following:

/opt/etc/init.d/S01syslog-ng stop  <--whiich stopped syslog-ng and wrote a stop message to the log
syslog-ng --preprocess-into "some file"  <--which wrote the full config to that file
scribe status <--which showed syslog-ng as dead
scribe restart <-- which restarted syslog-ng and wrote a start message to the log

So you are right, and my assumption was wrong. I think this is a clever thing; now that scribe has a reload to hot start a config change, should it hook to reload the ui?

Also, I discovered the reason I got funny log blocks in your earlier versions is because I'm loading the full scl blocks, and a few contain destinations that were being picked up in your earlier parsing. That's gone now.

I've asked @cmkelley about hooking uiScribe into scribe reloads etc. earlier today as it happens!

I've hopefully been clever enough now to only pull file( that appear on the next line after a destination declaration

 

[cmkelley]

I've asked @cmkelley about hooking uiScribe into scribe reloads etc. earlier today as it happens!

I've hopefully been clever enough now to only pull file( that appear on the next line after a destination declaration

Careful. Nothing prevents having more than 1 file in the destination, nor that the next line after 'destination {' is necessarily the file. You can't even necessarily count on the closing brace '}' being on a line all by itself to end the destination declaration. It's rather like a function in a shell script.

Not suggesting you need to solve all that right away, or that you necessarily even need to solve it, just be aware that as people get more experience with syslog-ng, the filters could get more complex.

 

[Jack Yaz]

Careful. Nothing prevents having more than 1 file in the destination, nor that the next line after 'destination {' is necessarily the file. You can't even necessarily count on the closing brace '}' being on a line all by itself to end the destination declaration. It's rather like a function in a shell script.

Not suggesting you need to solve all that right away, or that you necessarily even need to solve it, just be aware that as people get more experience with syslog-ng, the filters could get more complex.

Is it safe to assume that any file lines between destination{ and the next} are OK. Can additional {} be included in a destination declaration?

 

[cmkelley]

Is it safe to assume that any file lines between destination{ and the next} are OK. Can additional {} be included in a destination declaration?

Hrm. I'll have to have a look at some of the examples in the scl directory and see what they've done there. Those are likely to be more complex than people will use on the router anytime soon.

 

[elorimer]

Is it safe to assume that any file lines between destination{ and the next} are OK. Can additional {} be included in a destination declaration?

Technically it is between "{" and "};"
In between can be lots of stuff, on the same line or not but in there having

file("nameoffile");

The scl files have a number of block definitions, so for example in earlier iterations your code was picking up:

block source osquery(file("/var/log/osquery/osqueryd.results.log") prefix(".osquery.") ...) {
  channel {
    source { file("`file`" program-override("osquery") flags(no-parse) `__VARARGS__`); };
    parser { json-parser (prefix("`prefix`")); };
  };
};

and

block destination osquery(pipe("/var/osquery/syslog_pipe") ...) {
  channel {
    rewrite {
      set("$MESSAGE", value("ESCAPED_MESSAGE"));
      subst("\"","\"\"", value("ESCAPED_MESSAGE"), flags(global));
    };
    destination {
        pipe(`pipe` template(t_osquery));
        `__VARARGS__`;
    };
  };
};

So it is possible to have multiple braces. But what you have in the latest is no longer picking up any of the scl instances it shouldn't.

 

[Swistheater]

so far it is working great. great job jack! keep 'em rollin'

 

[Swistheater]

note any one who wants to follow jack on git hub click this link https://github.com/jackyaz
and click follow under his picture.. also on the main thread there is donations for those who love his work and want to support his efforts more.

 

[Swistheater]

so far it is working great. great job jack! keep 'em rollin'

My one question today is, not really important, but is there away to make the time on the clock appear more accurate, or is it going to be limited by the speed it takes to load with the script and custom gui running on top of it. Note: this is not a critical issue, just a cosmetic question.