Sep 24 14:30:01 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:30:03 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 4 on PIDS: 27512 Sep 24 14:30:04 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 7 on PIDS: 27512 Sep 24 14:30:11 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:30:13 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 5 on PIDS: 27512 Sep 24 14:30:14 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 8 on PIDS: 27512 Sep 24 14:30:21 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:30:24 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 6 on PIDS: 27512 Sep 24 14:30:24 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 9 on PIDS: 27512 Sep 24 14:30:31 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:30:34 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 7 on PIDS: 27512 Sep 24 14:30:34 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 10 on PIDS: 27512 Sep 24 14:30:41 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:30:44 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 8 on PIDS: 27512 Sep 24 14:30:44 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 11 on PIDS: 27512 Sep 24 14:30:51 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:30:54 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 9 on PIDS: 27512 Sep 24 14:30:54 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 12 on PIDS: 27512 Sep 24 14:31:01 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:31:04 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 10 on PIDS: 27512 Sep 24 14:31:04 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 13 on PIDS: 27512 Sep 24 14:31:11 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:31:14 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 11 on PIDS: 27512 Sep 24 14:31:14 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 14 on PIDS: 27512 Sep 24 14:31:21 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:31:24 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 12 on PIDS: 27512 Sep 24 14:31:24 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 15 on PIDS: 27512 Sep 24 14:31:31 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:31:34 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 13 on PIDS: 27512 Sep 24 14:31:34 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 16 on PIDS: 27512 Sep 24 14:31:41 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:31:44 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 14 on PIDS: 27512 Sep 24 14:31:44 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 17 on PIDS: 27512 Sep 24 14:31:51 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:31:54 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 15 on PIDS: 27512 Sep 24 14:31:54 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 18 on PIDS: 27512 Sep 24 14:32:01 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:32:04 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 16 on PIDS: 27512 Sep 24 14:32:04 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 19 on PIDS: 27512 Sep 24 14:32:11 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:32:14 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 17 on PIDS: 27512 Sep 24 14:32:14 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 20 on PIDS: 27512 Sep 24 14:32:21 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:32:24 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 18 on PIDS: 27512 Sep 24 14:32:24 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 21 on PIDS: 27512 Sep 24 14:32:31 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:32:34 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 19 on PIDS: 27512 Sep 24 14:32:35 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 22 on PIDS: 27512 Sep 24 14:32:41 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:32:44 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 20 on PIDS: 27512 Sep 24 14:32:45 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 23 on PIDS: 27512 Sep 24 14:32:51 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:32:54 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 21 on PIDS: 27512 Sep 24 14:32:55 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 24 on PIDS: 27512 Sep 24 14:33:01 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:33:04 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 22 on PIDS: 27512 Sep 24 14:33:05 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 25 on PIDS: 27512 Sep 24 14:33:11 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:33:14 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 23 on PIDS: 27512 Sep 24 14:33:15 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 26 on PIDS: 27512 Sep 24 14:33:21 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:33:25 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 24 on PIDS: 27512 Sep 24 14:33:25 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 27 on PIDS: 27512 Sep 24 14:33:31 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:33:35 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 25 on PIDS: 27512 Sep 24 14:33:35 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 28 on PIDS: 27512 Sep 24 14:33:41 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:33:45 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 26 on PIDS: 27512 Sep 24 14:33:45 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 29 on PIDS: 27512 Sep 24 14:33:51 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:33:55 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 27 on PIDS: 27512 Sep 24 14:33:55 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 30 on PIDS: 27512 Sep 24 14:34:01 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:34:05 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 28 on PIDS: 27512 Sep 24 14:34:05 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 31 on PIDS: 27512 Sep 24 14:34:11 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:34:15 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 29 on PIDS: 27512 Sep 24 14:34:15 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 32 on PIDS: 27512 Sep 24 14:34:21 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:34:25 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 30 on PIDS: 27512 Sep 24 14:34:25 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 33 on PIDS: 27512 Sep 24 14:34:31 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:34:35 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 31 on PIDS: 27512 Sep 24 14:34:35 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 34 on PIDS: 27512 Sep 24 14:34:41 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:34:45 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 32 on PIDS: 27512 Sep 24 14:34:45 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 35 on PIDS: 27512 Sep 24 14:34:51 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:34:55 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 33 on PIDS: 27512 Sep 24 14:34:55 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 36 on PIDS: 27512 Sep 24 14:35:01 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:35:05 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 34 on PIDS: 27512 Sep 24 14:35:05 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 37 on PIDS: 27512 Sep 24 14:35:11 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:35:15 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 35 on PIDS: 27512 Sep 24 14:35:15 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 38 on PIDS: 27512 Sep 24 14:35:21 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:35:25 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 36 on PIDS: 27512 Sep 24 14:35:25 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 39 on PIDS: 27512 Sep 24 14:35:31 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:35:35 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 37 on PIDS: 27512 Sep 24 14:35:36 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 40 on PIDS: 27512 Sep 24 14:35:41 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:35:45 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 38 on PIDS: 27512 Sep 24 14:35:46 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 41 on PIDS: 27512 Sep 24 14:35:51 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:35:55 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 39 on PIDS: 27512 Sep 24 14:35:56 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 42 on PIDS: 27512 Sep 24 14:36:01 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:36:05 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 40 on PIDS: 27512 Sep 24 14:36:06 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 43 on PIDS: 27512 Sep 24 14:36:11 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:36:15 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 41 on PIDS: 27512 Sep 24 14:36:16 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 44 on PIDS: 27512 Sep 24 14:36:21 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:36:26 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 42 on PIDS: 27512 Sep 24 14:36:26 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 45 on PIDS: 27512 Sep 24 14:36:32 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:36:36 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 43 on PIDS: 27512 Sep 24 14:36:36 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 46 on PIDS: 27512 Sep 24 14:36:42 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:36:46 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 44 on PIDS: 27512 Sep 24 14:36:46 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 47 on PIDS: 27512 Sep 24 14:36:52 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:36:56 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 45 on PIDS: 27512 Sep 24 14:36:56 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 48 on PIDS: 27512 Sep 24 14:37:02 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:37:06 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 46 on PIDS: 27512 Sep 24 14:37:06 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 49 on PIDS: 27512 Sep 24 14:37:12 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:37:16 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 47 on PIDS: 27512 Sep 24 14:37:16 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 50 on PIDS: 27512 Sep 24 14:37:22 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:37:26 xen1 ha-lizard: 27510 ha-lizard already running: Attempt 48 on PIDS: 27512 Sep 24 14:37:26 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 51 on PIDS: 27512 Sep 24 14:37:26 xen1 ha-lizard: 26438 ha-lizard failed to spawn new instance after 51 attmepts. Check xen1.xenpool.local for possible hung process Sep 24 14:37:26 xen1 ha-lizard: 26438 MONITOR has reached configured threshhold - 50 - MONITOR_KILLALL is enabled. - attempting to kill all existing processes Sep 24 14:37:26 xen1 ha-lizard: 26438 Successfully killed all instances of ha-lizard.sh Sep 24 14:37:32 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:37:36 xen1 ha-lizard: 27510 Spawning new instance of ha-lizard Sep 24 14:37:36 xen1 ha-lizard: 9579 Checking if this host is a Pool Master or Slave Sep 24 14:37:36 xen1 ha-lizard: 9579 This host's pool status = master Sep 24 14:37:36 xen1 ha-lizard: 9579 Checking if ha-lizard is enabled for this pool Sep 24 14:37:36 xen1 ha-lizard: 9579 check_ha_enabled: Checking if ha-lizard is enabled for pool: 43974b5c-6bc8-5030-a990-def1bd87f683 Sep 24 14:37:36 xen1 ha-lizard: 9579 check_ha_enabled: ha-lizard is enabled Sep 24 14:37:36 xen1 ha-lizard: 9579 ha-lizard is enabled Sep 24 14:37:36 xen1 ha-lizard: 9579 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Sep 24 14:37:36 xen1 ha-lizard: 9579 DISABLED_VAPPS=(31b5e105-7d27-d752-db38-f8c5a93852af) ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=1 FENCE_HEURISTICS_IPS=() FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=1 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=0 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=45 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=50 MONITOR_SCANRATE=10 OP_MODE=1 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=1 XAPI_COUNT=5 XAPI_DELAY=15 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Sep 24 14:37:36 xen1 ha-lizard: 9579 This host detected as pool Master Sep 24 14:37:36 xen1 ha-lizard: 9579 Found 2 hosts in pool Sep 24 14:37:36 xen1 ha-lizard: 9579 Calling function write_pool_state Sep 24 14:37:36 xen1 ha-lizard: 9659 Calling function autoselect_slave Sep 24 14:37:36 xen1 ha-lizard: 9666 Calling function check_slave_status Sep 24 14:37:36 xen1 ha-lizard: 26438 ha-lizard already running: Attempt 52 on PIDS: 9662 9661 9659 9579 9574 9573 9571 Sep 24 14:37:36 xen1 ha-lizard: 26438 ha-lizard failed to spawn new instance after 52 attmepts. Check xen1.xenpool.local for possible hung process Sep 24 14:37:36 xen1 ha-lizard: 26438 MONITOR has reached configured threshhold - 50 - MONITOR_KILLALL is enabled. - attempting to kill all existing processes Sep 24 14:37:36 xen1 ha-lizard: 9666 get_pool_host_list: returned f71545f5-e8e6-4cc8-9130-89d67992b735 de769f75-ac15-4c1f-8be9-24e457319c55 Sep 24 14:37:36 xen1 ha-lizard: 9659 autoselect_slave: This host UUID found: de769f75-ac15-4c1f-8be9-24e457319c55 Sep 24 14:37:36 xen1 ha-lizard: 26438 Successfully killed all instances of ha-lizard.sh Sep 24 14:37:42 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:37:46 xen1 ha-lizard: 26438 Spawning new instance of ha-lizard Sep 24 14:37:46 xen1 ha-lizard: 9981 Checking if this host is a Pool Master or Slave Sep 24 14:37:46 xen1 ha-lizard: 9981 This host's pool status = master Sep 24 14:37:46 xen1 ha-lizard: 9981 Checking if ha-lizard is enabled for this pool Sep 24 14:37:46 xen1 ha-lizard: 9981 check_ha_enabled: Checking if ha-lizard is enabled for pool: 43974b5c-6bc8-5030-a990-def1bd87f683 Sep 24 14:37:46 xen1 ha-lizard: 9981 check_ha_enabled: ha-lizard is enabled Sep 24 14:37:46 xen1 ha-lizard: 9981 ha-lizard is enabled Sep 24 14:37:47 xen1 ha-lizard: 9981 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Sep 24 14:37:47 xen1 ha-lizard: 9981 DISABLED_VAPPS=(31b5e105-7d27-d752-db38-f8c5a93852af) ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=1 FENCE_HEURISTICS_IPS=() FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=1 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=0 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=45 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=50 MONITOR_SCANRATE=10 OP_MODE=1 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=1 XAPI_COUNT=5 XAPI_DELAY=15 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Sep 24 14:37:47 xen1 ha-lizard: 9981 This host detected as pool Master Sep 24 14:37:47 xen1 ha-lizard: 9981 Found 2 hosts in pool Sep 24 14:37:47 xen1 ha-lizard: 9981 Calling function write_pool_state Sep 24 14:37:47 xen1 ha-lizard: 10060 Calling function autoselect_slave Sep 24 14:37:47 xen1 ha-lizard: 10067 Calling function check_slave_status Sep 24 14:37:47 xen1 ha-lizard: 10060 autoselect_slave: This host UUID found: de769f75-ac15-4c1f-8be9-24e457319c55 Sep 24 14:37:47 xen1 ha-lizard: 10067 get_pool_host_list: returned f71545f5-e8e6-4cc8-9130-89d67992b735 de769f75-ac15-4c1f-8be9-24e457319c55 Sep 24 14:37:47 xen1 ha-lizard: 10060 autoselect_slave: MASTER host UUID found: de769f75-ac15-4c1f-8be9-24e457319c55 Sep 24 14:37:47 xen1 ha-lizard: 10060 autoselect_slave: de769f75-ac15-4c1f-8be9-24e457319c55 is Master UUID - excluding from list of available slaves Sep 24 14:37:47 xen1 ha-lizard: 10067 check_slave_status: Removing Master UUID from list of Hosts Sep 24 14:37:47 xen1 ha-lizard: 9981 get_vms_on_host: Returned 9ecfe259-5dee-8ab7-789e-fca5ebcc2795 4f8e9022-4792-af51-b008-35a69f4504b4 2e2860b5-8e28-f33f-0ec6-db0b4053b9d3 Sep 24 14:37:47 xen1 ha-lizard: 10060 autoselect_slave: 1 available Slave UUIDs found: f71545f5-e8e6-4cc8-9130-89d67992b735 Sep 24 14:37:47 xen1 ha-lizard: 9981 get_vms_on_host: Returned 89d5824d-f56c-91c0-793d-ca1cdb7b46fd f6e5d909-d19c-4299-e42d-d1c11ec82a6c 58e722f9-daea-97f4-65d5-7d8271d569ae Sep 24 14:37:47 xen1 ha-lizard: 10067 get_pool_ip_list: returned 10.10.10.243 Sep 24 14:37:47 xen1 ha-lizard: 10060 autoselect_slave: Selected Slave: f71545f5-e8e6-4cc8-9130-89d67992b735 = Current slave: f71545f5-e8e6-4cc8-9130-89d67992b735 - ignoring update Sep 24 14:37:47 xen1 ha-lizard: 10067 check_xapi: Pool Host 10.10.10.243 xapi status = 1 Sep 24 14:37:47 xen1 ha-lizard: 10067 check_slave_status: Host IP Address check Status Array for Slaves = (1) Sep 24 14:37:47 xen1 ha-lizard: 10067 check_slave_status: Failed slave count = 0 Sep 24 14:37:47 xen1 ha-lizard: 10067 check_slave_status: No Failed slaves detected Sep 24 14:37:47 xen1 ha-lizard: 10067 Function check_slave_status reported no failures: calling vm_mon Sep 24 14:37:47 xen1 ha-lizard: 10067 vm_mon: ha-lizard in operating mode 1 - managing pool appliances Sep 24 14:37:47 xen1 ha-lizard: 9981 check_ha_enabled: Checking if ha-lizard is enabled for pool: 43974b5c-6bc8-5030-a990-def1bd87f683 Sep 24 14:37:47 xen1 ha-lizard: 9981 check_ha_enabled: ha-lizard is enabled Sep 24 14:37:47 xen1 ha-lizard: 10067 vm_mon: 1 vapps found: 31b5e105-7d27-d752-db38-f8c5a93852af a132f9d0-b7f4-fa64-085c-466dc602b992 Sep 24 14:37:47 xen1 ha-lizard: 10067 vm_mon: Checking if ha_lizard is enabled for UUID: 31b5e105-7d27-d752-db38-f8c5a93852af a132f9d0-b7f4-fa64-085c-466dc602b992 Sep 24 14:37:47 xen1 ha-lizard: 10067 vm_mon: HA disabled for UUID: 31b5e105-7d27-d752-db38-f8c5a93852af a132f9d0-b7f4-fa64-085c-466dc602b992 Sep 24 14:37:47 xen1 ha-lizard: 9981 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Sep 24 14:37:47 xen1 ha-lizard: 9981 get_pool_host_list: returned f71545f5-e8e6-4cc8-9130-89d67992b735 de769f75-ac15-4c1f-8be9-24e457319c55 Sep 24 14:37:47 xen1 ha-lizard: 9981 get_pool_ip_list: returned 10.10.10.243 Sep 24 14:37:47 xen1 ha-lizard: 9981 get_pool_ip_list: returned 10.10.10.243 10.10.10.242 Sep 24 14:37:52 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:38:02 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:38:12 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:38:21 xen1 ha-lizard: 9571 Spawning new instance of ha-lizard Sep 24 14:38:21 xen1 ha-lizard: 11165 Checking if this host is a Pool Master or Slave Sep 24 14:38:21 xen1 ha-lizard: 11165 This host's pool status = master Sep 24 14:38:21 xen1 ha-lizard: 11165 Checking if ha-lizard is enabled for this pool Sep 24 14:38:21 xen1 ha-lizard: 11165 check_ha_enabled: Checking if ha-lizard is enabled for pool: 43974b5c-6bc8-5030-a990-def1bd87f683 Sep 24 14:38:21 xen1 ha-lizard: 11165 check_ha_enabled: ha-lizard is enabled Sep 24 14:38:21 xen1 ha-lizard: 11165 ha-lizard is enabled Sep 24 14:38:21 xen1 ha-lizard: 11165 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Sep 24 14:38:21 xen1 ha-lizard: 11165 DISABLED_VAPPS=(31b5e105-7d27-d752-db38-f8c5a93852af) ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=1 FENCE_HEURISTICS_IPS=() FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=1 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=0 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=45 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=50 MONITOR_SCANRATE=10 OP_MODE=1 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=1 XAPI_COUNT=5 XAPI_DELAY=15 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Sep 24 14:38:21 xen1 ha-lizard: 11165 This host detected as pool Master Sep 24 14:38:21 xen1 ha-lizard: 11165 Found 2 hosts in pool Sep 24 14:38:21 xen1 ha-lizard: 11165 Calling function write_pool_state Sep 24 14:38:21 xen1 ha-lizard: 11244 Calling function autoselect_slave Sep 24 14:38:21 xen1 ha-lizard: 11250 Calling function check_slave_status Sep 24 14:38:21 xen1 ha-lizard: 11244 autoselect_slave: This host UUID found: de769f75-ac15-4c1f-8be9-24e457319c55 Sep 24 14:38:21 xen1 ha-lizard: 11250 get_pool_host_list: returned f71545f5-e8e6-4cc8-9130-89d67992b735 de769f75-ac15-4c1f-8be9-24e457319c55 Sep 24 14:38:21 xen1 ha-lizard: 11244 autoselect_slave: MASTER host UUID found: de769f75-ac15-4c1f-8be9-24e457319c55 Sep 24 14:38:22 xen1 ha-lizard: 11165 get_vms_on_host: Returned 9ecfe259-5dee-8ab7-789e-fca5ebcc2795 4f8e9022-4792-af51-b008-35a69f4504b4 2e2860b5-8e28-f33f-0ec6-db0b4053b9d3 Sep 24 14:38:22 xen1 ha-lizard: 11244 autoselect_slave: de769f75-ac15-4c1f-8be9-24e457319c55 is Master UUID - excluding from list of available slaves Sep 24 14:38:22 xen1 ha-lizard: 11250 check_slave_status: Removing Master UUID from list of Hosts Sep 24 14:38:22 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:38:22 xen1 ha-lizard: 11165 get_vms_on_host: Returned 89d5824d-f56c-91c0-793d-ca1cdb7b46fd f6e5d909-d19c-4299-e42d-d1c11ec82a6c 58e722f9-daea-97f4-65d5-7d8271d569ae Sep 24 14:38:22 xen1 ha-lizard: 11244 autoselect_slave: 1 available Slave UUIDs found: f71545f5-e8e6-4cc8-9130-89d67992b735 Sep 24 14:38:22 xen1 ha-lizard: 11250 get_pool_ip_list: returned 10.10.10.243 Sep 24 14:38:22 xen1 ha-lizard: 11244 autoselect_slave: Selected Slave: f71545f5-e8e6-4cc8-9130-89d67992b735 = Current slave: f71545f5-e8e6-4cc8-9130-89d67992b735 - ignoring update Sep 24 14:38:22 xen1 ha-lizard: 11250 check_xapi: Pool Host 10.10.10.243 xapi status = 1 Sep 24 14:38:22 xen1 ha-lizard: 11250 check_slave_status: Host IP Address check Status Array for Slaves = (1) Sep 24 14:38:22 xen1 ha-lizard: 11250 check_slave_status: Failed slave count = 0 Sep 24 14:38:22 xen1 ha-lizard: 11250 check_slave_status: No Failed slaves detected Sep 24 14:38:22 xen1 ha-lizard: 11250 Function check_slave_status reported no failures: calling vm_mon Sep 24 14:38:22 xen1 ha-lizard: 11250 vm_mon: ha-lizard in operating mode 1 - managing pool appliances Sep 24 14:38:22 xen1 ha-lizard: 11165 check_ha_enabled: Checking if ha-lizard is enabled for pool: 43974b5c-6bc8-5030-a990-def1bd87f683 Sep 24 14:38:22 xen1 ha-lizard: 11165 check_ha_enabled: ha-lizard is enabled Sep 24 14:38:22 xen1 ha-lizard: 11250 vm_mon: 1 vapps found: 31b5e105-7d27-d752-db38-f8c5a93852af a132f9d0-b7f4-fa64-085c-466dc602b992 Sep 24 14:38:22 xen1 ha-lizard: 11250 vm_mon: Checking if ha_lizard is enabled for UUID: 31b5e105-7d27-d752-db38-f8c5a93852af a132f9d0-b7f4-fa64-085c-466dc602b992 Sep 24 14:38:22 xen1 ha-lizard: 11250 vm_mon: HA disabled for UUID: 31b5e105-7d27-d752-db38-f8c5a93852af a132f9d0-b7f4-fa64-085c-466dc602b992 Sep 24 14:38:22 xen1 ha-lizard: 11165 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Sep 24 14:38:22 xen1 ha-lizard: 11165 get_pool_host_list: returned f71545f5-e8e6-4cc8-9130-89d67992b735 de769f75-ac15-4c1f-8be9-24e457319c55 Sep 24 14:38:22 xen1 ha-lizard: 11165 get_pool_ip_list: returned 10.10.10.243 Sep 24 14:38:22 xen1 ha-lizard: 11165 get_pool_ip_list: returned 10.10.10.243 10.10.10.242 Sep 24 14:38:31 xen1 ha-lizard: 9972 ha-lizard already running: Attempt 1 on PIDS: 11158 Sep 24 14:38:32 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:38:41 xen1 ha-lizard: 9972 ha-lizard already running: Attempt 2 on PIDS: 11158 Sep 24 14:38:42 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:38:51 xen1 ha-lizard: 9972 ha-lizard already running: Attempt 3 on PIDS: 11158 Sep 24 14:38:52 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:39:02 xen1 ha-lizard: 9972 ha-lizard already running: Attempt 4 on PIDS: 11158 Sep 24 14:39:02 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:39:06 xen1 ha-lizard: 11156 ha-lizard already running: Attempt 1 on PIDS: 11158 Sep 24 14:39:12 xen1 ha-lizard: 9972 ha-lizard already running: Attempt 5 on PIDS: 11158 Sep 24 14:39:12 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:39:16 xen1 ha-lizard: 11156 ha-lizard already running: Attempt 2 on PIDS: 11158 Sep 24 14:39:22 xen1 ha-lizard: 9972 ha-lizard already running: Attempt 6 on PIDS: 11158 Sep 24 14:39:22 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:39:26 xen1 ha-lizard: 11156 ha-lizard already running: Attempt 3 on PIDS: 11158 Sep 24 14:39:32 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:39:32 xen1 ha-lizard: 9972 ha-lizard already running: Attempt 7 on PIDS: 11158 Sep 24 14:39:36 xen1 ha-lizard: 11156 ha-lizard already running: Attempt 4 on PIDS: 11158 Sep 24 14:39:42 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:39:42 xen1 ha-lizard: 9972 ha-lizard already running: Attempt 8 on PIDS: 11158 Sep 24 14:39:46 xen1 ha-lizard: 11156 ha-lizard already running: Attempt 5 on PIDS: 11158 Sep 24 14:39:52 xen1 ha-lizard: 8734 ha-lizard Watchdog: ha-lizard running - OK Sep 24 14:39:52 xen1 ha-lizard: 9972 ha-lizard already running: Attempt 9 on PIDS: 11158 Sep 24 14:39:56 xen1 ha-lizard: 11156 ha-lizard already running: Attempt 6 on PIDS: 11158