Forum
Welcome, Guest
Username: Password: Remember me
  • Page:
  • 1

TOPIC:

too many loggers 10 years 7 months ago #65

  • himiq
  • himiq's Avatar Topic Author
I has install ha-lizard v.1.6.42.3 on pool (2 XCP servers).
XCP release 1.6.10-61809c

After 2-3 hours on slave XCP server:
# ps axuwww | grep logger | grep ha-lizard | grep -v grep | wc -l
105

After 6+ hours quantity of same processes was 400 and more.

Processes (all same except running time):
# ps axuwww | grep logger | grep ha-lizard | grep -v grep
root 32158 0.0 0.0 3796 468 ? S 09:51 0:00 logger -t ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh
...

But on same time om master XCP server processes:
# ps axuwww | grep logger | grep ha-lizard | grep -v grep | wc -l
10
(time to time 9 or 11 but no more)

Whats wrong ?

Please Log in or Create an account to join the conversation.

too many loggers 10 years 7 months ago #66

There should only be a small number of these processes under normal conditions. Is it possible there is a recurring error in your pool causing a large number of logger threads? Can you check the log file output while you see a large number of logger threads and check for any error messages? They should be easily seen since they will be prefixed with "NOTICE"

You can view the log with:

ha-cfg log
or
tail -f /var/log/messages | grep ha-lizard

Please post your finding and any other details about your setup so we can try to reproduce.
Thank You

Please Log in or Create an account to join the conversation.

too many loggers 10 years 7 months ago #81

  • himiq
  • himiq's Avatar Topic Author
after 18 hours from restart xcp-servers:
on "master" (xen1) :
# ps axuwww | grep logger | grep ha-lizard | wc -l
245
on "slave" (xen2) :
# ps axuwww | grep logger | grep ha-lizard | wc -l
229

on both servers type of procecces are identical, like this:
oot 32631 0.0 0.0 3796 468 ? S Sep23 0:00 logger -t ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh
root 8669 0.0 0.0 3796 576 ? S Sep23 0:00 logger -t ha-lizard-NOTICE-/etc/rc3.d/S99ha-lizard
root 8739 0.0 0.0 3796 468 ? S Sep23 0:00 logger -t ha-lizard-NOTICE-/etc/rc3.d/S99ha-lizard-watchdog

in attachment - config and logs from both server for same time period.

Thanks in advance
Attachments:

Please Log in or Create an account to join the conversation.

too many loggers 10 years 7 months ago #82

Thanks for posting the logs. In reviewing the logs it looks like KILLALL is being triggered repeatedly. This is a safetey mechanism to ensure that HA processes never lock or run for too long. So, after 50 tries, any dangling processes are killed and a new instance of HA-Lizard is spawned. So, there are 2 issues here:

1 - the killall function needs to be updated to cleanup any dangling logger processes. Each time the service is started 2 loggers are initiated and then not cleaned up when killal is triggered. We will post an update in a few days that properly cleans up dangling logger processes. That will resolve the issue with too many loggers.

2 - killall should rarely be triggered. We will try to recreate your scenario to see what the underlying problem can be that is causing it to trigger repeatedly.

Please Log in or Create an account to join the conversation.

too many loggers 10 years 6 months ago #107

We have been unable to reproduce the logger issue. In looking at your logs, it appears that somehow the ha-lizard service is running in duplicate which causes repreated restarts after some time. Did you by any chance start the service and also manually run a script in /etc/ha-lizard/init/ ?

Can you try 'service ha-lizaard stop -w' and then 'ps aux | grep ha-lizard'. There should be no processes.

Please Log in or Create an account to join the conversation.

  • Page:
  • 1