Defunct cfg_server Zombies

[SomeWhereOverTheRainBow]

All:

I ended up modifying the zomg_cfg_server script (with a 1 minute cronjob) to only kill -9 any newly created cfg_server processes greater than the original 3 processes, which appears to be working without invoking the watchdog start_cfgsync process (as the parent cfg_server is preserved in a healthy state), subverting the restart_wireless processes on the AiMesh Nodes, and avoiding constant writing to the primary router's syslog.

# cat /jffs/sbin/zomg_cfg_server
#!/bin/sh

i=1

procs="$(pidof cfg_server)"

for pid in $procs; do
   if [ "$i" -gt "3" ]; then
      #echo $i
      #echo $pid
      /usr/bin/kill -9 $pid
      /usr/bin/logger "Running /jffs/sbin/zomg_cfg_server"
   fi

   i=$((i + 1))
done

It seems to be working very well. I'll update this post, if there are any observed ill effects.

I believe this to be a more ideal solution than the previous killall recommendation.

Respectfully,


Gary

glad you found something that works.

It would be cool if something like this would work.

#!/bin/sh
procs="$(ps -T | awk '/cfg_server/ {print $1}' | awk '{printf "%s ",$0} END {print ""}')"
if [ "$procs" ]; then
  for PID in $procs; do
    if awk '{ print }' "/proc/${PID}/cmdline" | grep -q defunct; then
      kill -9 "$PID"
      break
    fi
  done
fi

 

[garycnew]

glad you found something that works.

It would be cool if something like this would work.

#!/bin/sh
procs="$(ps -T | awk '/cfg_server/ {print $1}' | awk '{printf "%s ",$0} END {print ""}')"
if [ "$procs" ]; then
  for PID in $procs; do
    if awk '{ print }' "/proc/${PID}/cmdline" | grep -q defunct; then
      kill -9 "$PID"
      break
    fi
  done
fi

@SomeWhereOverTheRainBow

Unfortunately, I believe, if the process gets to the defunct state, it's too late to kill it.

As always, I appreciate your skills.

Respectfully,


Gary

 

[SomeWhereOverTheRainBow]

@SomeWhereOverTheRainBow

Unfortunately, I believe, if the process gets to the defunct state, it's too late to kill it.

As always, I appreciate your skills.

Respectfully,


Gary

kill -s SIGCHLD $PID

tells the parent process to kill and cleanup any zombie processes. If that fails, the

kill -9 is the only other alternative.

 

[garycnew]

kill -s SIGCHLD $PID

tells the parent process to kill and cleanup any zombie processes. If that fails, the

kill -9 is the only other alternative.

Nice! I'll have to give that a try. Would the $PID be the parent or the child?

Previously, kill -9 only worked on the parent.

 

[SomeWhereOverTheRainBow]

Nice! I'll have to give that a try. Would the $PID be the parent or the child?

Previously, kill -9 only worked on the parent.

In your case, you would be using $PID to the parent. The hope that you would tell the parent to kill the defunct processes, otherwise the alternative is to kill -9 $PID the parent. ( if not instantly) At somepoint the zombie would hopefully cease to be a process because INIT would eventually kill them.

 

[garycnew]

Note: The reciprocating processes on the AiMesh Nodes (cfg_client) do not appear to go defunct running the zomg_cfg_server on the Primary Router.

The modified zomg_cfg_server script is still working well.

 

[SomeWhereOverTheRainBow]

All:

I ended up modifying the zomg_cfg_server script (with a 1 minute cronjob) to only kill -9 any newly created cfg_server processes greater than the original 3 processes, which appears to be working without invoking the watchdog start_cfgsync process (as the parent cfg_server is preserved in a healthy state), subverting the restart_wireless processes on the AiMesh Nodes, and avoiding constant writing to the primary router's syslog.

# cat /jffs/sbin/zomg_cfg_server
#!/bin/sh

i=1

procs="$(pidof cfg_server)"

for pid in $procs; do
   if [ "$i" -gt "3" ]; then
      #echo $i
      #echo $pid
      /usr/bin/kill -9 $pid
      /usr/bin/logger "Running /jffs/sbin/zomg_cfg_server"
   fi

   i=$((i + 1))
done

It seems to be working very well. I'll update this post, if there are any observed ill effects.

I believe this to be a more ideal solution than the previous killall recommendation.

Respectfully,


Gary

Is this the version you are using still?

Have you notice any logs when it indicates it killed an extra process?

 

[garycnew]

Is this the version you are using still?

Have you notice any logs when it indicates it killed an extra process?

@SomeWhereOverTheRainBow

Correct... I'm still using the referenced version of the modified zomg_cfg_server script.

I've specifically made note that there are no log entries generated, after the script indicates an extra process has been killed.

I haven't encounter the previous swapping issues, either. It seems solid!

Respectfully,


Gary

 

[SomeWhereOverTheRainBow]

@SomeWhereOverTheRainBow

Correct... I'm still using the referenced version of the modified zomg_cfg_server script.

I've specifically made note that there are no log entries generated, after the script indicates an extra process has been killed.

I haven't encounter the previous swapping issues, either. It seems solid!

Respectfully,


Gary

Do you run it from a cronjob? Or do you invoke it from service event?

 

[garycnew]

Do you run it from a cronjob? Or do you invoke it from service event?

Presently, I have it configured as a cronjob, which is executed every minute.

I've considered configuring it similarly as the original zomg_cfg_server script with a while loop, sleep, and then reference it in the post-mount script to produce a less than one minute interval.

Are you just curious or are you facing a similar issue?

 

[SomeWhereOverTheRainBow]

Presently, I have it configured as a cronjob, which is executed every minute.

I've considered configuring it similarly as the original zomg_cfg_server script with a while loop, sleep, and then reference it in the post-mount script to produce a less than one minute interval.

Are you just curious or are you facing a similar issue?

You can call it off of the service event for cfg_server

 

[SomeWhereOverTheRainBow]

Presently, I have it configured as a cronjob, which is executed every minute.

I've considered configuring it similarly as the original zomg_cfg_server script with a while loop, sleep, and then reference it in the post-mount script to produce a less than one minute interval.

Are you just curious or are you facing a similar issue?

create (or use ) /jffs/scripts/service_event_end

add this one liner to it.

if echo "$2" | /bin/grep -q "^cfgsync"; then { sh /jffs/sbin/zomg_cfg_server & }; fi # Zomg_cfg_sync

chmod 755 /jffs/scripts/service_event_end

The event will be called any time there is a cfg sync.

That should free up you having to rely on using cron-job and should eliminate needing to run it in a loop.

 

[garycnew]

create (or use ) /jffs/scripts/service_event_end

add this one liner to it.

if echo "$2" | /bin/grep -q "^cfgsync"; then { sh /jffs/sbin/zomg_cfg_server & }; fi # Zomg_cfg_sync

chmod 755 /jffs/scripts/service_event_end

The event will be called any time there is a cfg sync.

That should free up you having to rely on using cron-job and should eliminate needing to run it in a loop.

Nice! I'd prefer a solution that's more responsive.

I'll try to implement it, when I have a chance.

Thanks, again!

 

[SomeWhereOverTheRainBow]

Nice! I'd prefer a solution that's more responsive.

I'll try to implement it, when I have a chance.

Thanks, again!

When you try to incorporate the service event, You may want to add a small while loop where the zomg server sleeps until the procs count is greater than 3, also allow the script to exit if there are multiple runs of zomg_cfg server detected.

#!/bin/sh

i=1

pid_loc="pidof $(basename "$0")"
procs="pidof cfg_server"

if [ "$(printf "%s" "$($pid_loc)" | wc -w)" -gt 1 ]; then exit; fi
while [ "$(printf "%s" "$($procs)" | wc -w)" -le 3 ]; do sleep 1; done

for pid in $($procs); do
   if [ "$i" -gt "3" ]; then
      #echo $i
      #echo $pid
      /usr/bin/kill -9 $pid
      /usr/bin/logger "Running /jffs/sbin/zomg_cfg_server"
   fi
   i=$((i + 1))
done

 

[Martinski]

while [ "$(printf "%s" "$procs" | wc -w)" -le 3 ]; do sleep 1; done

The above "while-loop" becomes an endless loop when the number of process IDs <= 3 because the value of $procs is never updated WITHIN the loop itself so it will never be greater than 3 in that case.

One solution would be:

while [ "$(printf "%s" "$(pidof cfg_server)" | wc -w)" -le 3 ]; do sleep 1; done

## Moved line AFTER the while-loop and BEFORE the for-loop ##
procs="$(pidof cfg_server)"

 

[SomeWhereOverTheRainBow]

The above "while-loop" becomes an endless loop when the number of process IDs <= 3 because the value of $procs is never updated WITHIN the loop itself so it will never be greater than 3 in that case.

One solution would be:

while [ "$(printf "%s" "$(pidof cfg_server)" | wc -w)" -le 3 ]; do sleep 1; done

## Moved line AFTER the while-loop and BEFORE the for-loop ##
procs="$(pidof cfg_server)"

or change it to

#!/bin/sh

i=1

pid_loc="pidof $(basename "$0")"
procs="pidof cfg_server"

if [ "$(printf "%s" "$($pid_loc)" | wc -w)" -gt 1 ]; then exit; fi
while [ "$(printf "%s" "$($procs)" | wc -w)" -le 3 ]; do sleep 1; done

for pid in $($procs); do
   if [ "$i" -gt "3" ]; then
      #echo $i
      #echo $pid
      /usr/bin/kill -9 $pid
      /usr/bin/logger "Running /jffs/sbin/zomg_cfg_server"
   fi
   i=$((i + 1))
done

or reiterate

procs="$(pidof cfg_server)"

within the loop itself after the sleep.

 

[garycnew]

Is it just me or does it seem like everyone on the Asuswrt-Merlin forum is shell script l33t?

Thanks, again, for the assistance!

Respectfully,


Gary

 

[Martinski]

...
while [ "$(printf "%s" "$($procs)" | wc -w)" -le 3 ]; do sleep 1; done
...

I'm not sure if that particular syntax $($procs) actually works in the router's shell. It may since I don't know the full capabilities & limitations of the router's interpreter. I know it doesn't work in standard Linux bash (I'm currently using Ubuntu 20.04 LTS).

 

[SomeWhereOverTheRainBow]

I'm not sure if that particular syntax $($procs) actually works in the router's shell. It may since I don't know the full capabilities & limitations of the router's interpreter. I know it doesn't work in standard Linux bash (I'm currently using Ubuntu 20.04 LTS).

it does I tested it. You can test it with simple echo inside the terminal. when dealing with numbers that is Pidof returns numbers and we are telling it to execute a legit command.

 

[SomeWhereOverTheRainBow]

all the $($procs) does is saying is let us execute the command at the time it is called. If we were using an actual value that could not be executed it wouldn't work.