This will display a real time log session for ha-lizard Control + C to exit. Jul 23 09:27:27 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:27:30 HA-xen001 ha-lizard: 20779 Spawning new instance of ha-lizard Jul 23 09:27:30 HA-xen001 ha-lizard: 20779 check_logger_processes: Checking logger processes Jul 23 09:27:30 HA-xen001 ha-lizard: 20779 check_logger_processes: No processes to clear Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 This iteration is count 202 Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 Checking if this host is a Pool Master or Slave Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 This host's pool status = master Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 Checking if ha-lizard is enabled for this pool Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 check_ha_enabled: ha-lizard is enabled Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 ha-lizard is enabled Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 This host detected as pool Master Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 Found 2 hosts in pool Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 Calling function write_pool_state Jul 23 09:27:30 HA-xen001 ha-lizard: 21508 Calling function autoselect_slave Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 Calling function check_slave_status Jul 23 09:27:30 HA-xen001 ha-lizard: 21508 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:27:30 HA-xen001 ha-lizard: 21508 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 check_slave_status: Removing Master UUID from list of Hosts Jul 23 09:27:30 HA-xen001 ha-lizard: 21508 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves Jul 23 09:27:30 HA-xen001 ha-lizard: 21508 autoselect_slave: 1 available Slave UUIDs found: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:27:30 HA-xen001 ha-lizard: 21508 autoselect_slave: Selected Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 = Current slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 - ignoring update Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 check_xapi: Pool Host 192.168.1.61 xapi status = 1 Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 check_slave_status: Host IP Address check Status Array for Slaves = (1) Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 check_slave_status: Failed slave count = 0 Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 check_slave_status: No Failed slaves detected Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 Function check_slave_status reported no failures: calling vm_mon Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 check_ha_enabled: ha-lizard is enabled Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 vm_mon: Removing Control Domains from VM list Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running Jul 23 09:27:30 HA-xen001 ha-lizard: 21442 get_pool_ip_list: returned 192.168.1.61 192.168.1.60 Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running Jul 23 09:27:30 HA-xen001 ha-lizard: 21516 vm_mon: 0 Eligible Halted VMs found Jul 23 09:27:37 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:27:45 HA-xen001 ha-lizard: 21434 Spawning new instance of ha-lizard Jul 23 09:27:45 HA-xen001 ha-lizard: 21434 check_logger_processes: Checking logger processes Jul 23 09:27:45 HA-xen001 ha-lizard: 21434 check_logger_processes: No processes to clear Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 This iteration is count 203 Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 Checking if this host is a Pool Master or Slave Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 This host's pool status = master Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 Checking if ha-lizard is enabled for this pool Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 check_ha_enabled: ha-lizard is enabled Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 ha-lizard is enabled Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 This host detected as pool Master Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 Found 2 hosts in pool Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 Calling function write_pool_state Jul 23 09:27:45 HA-xen001 ha-lizard: 22438 Calling function autoselect_slave Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 Calling function check_slave_status Jul 23 09:27:45 HA-xen001 ha-lizard: 22438 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:27:45 HA-xen001 ha-lizard: 22438 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 check_slave_status: Removing Master UUID from list of Hosts Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb Jul 23 09:27:45 HA-xen001 ha-lizard: 22438 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves Jul 23 09:27:45 HA-xen001 ha-lizard: 22438 autoselect_slave: 1 available Slave UUIDs found: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:27:45 HA-xen001 ha-lizard: 22438 autoselect_slave: Selected Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 = Current slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 - ignoring update Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 check_xapi: Pool Host 192.168.1.61 xapi status = 1 Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 check_slave_status: Host IP Address check Status Array for Slaves = (1) Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 check_slave_status: Failed slave count = 0 Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 check_slave_status: No Failed slaves detected Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 Function check_slave_status reported no failures: calling vm_mon Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 check_ha_enabled: ha-lizard is enabled Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 vm_mon: Removing Control Domains from VM list Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running Jul 23 09:27:45 HA-xen001 ha-lizard: 22445 vm_mon: 0 Eligible Halted VMs found Jul 23 09:27:45 HA-xen001 ha-lizard: 22374 get_pool_ip_list: returned 192.168.1.61 192.168.1.60 Jul 23 09:27:47 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:27:57 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:28:00 HA-xen001 ha-lizard: 22366 Spawning new instance of ha-lizard Jul 23 09:28:00 HA-xen001 ha-lizard: 22366 check_logger_processes: Checking logger processes Jul 23 09:28:00 HA-xen001 ha-lizard: 22366 check_logger_processes: No processes to clear Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 This iteration is count 204 Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 Checking if this host is a Pool Master or Slave Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 This host's pool status = master Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 Checking if ha-lizard is enabled for this pool Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 check_ha_enabled: ha-lizard is enabled Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 ha-lizard is enabled Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 This host detected as pool Master Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 Found 2 hosts in pool Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 Calling function write_pool_state Jul 23 09:28:00 HA-xen001 ha-lizard: 23077 Calling function autoselect_slave Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 Calling function check_slave_status Jul 23 09:28:00 HA-xen001 ha-lizard: 23077 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:00 HA-xen001 ha-lizard: 23077 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 check_slave_status: Removing Master UUID from list of Hosts Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb Jul 23 09:28:00 HA-xen001 ha-lizard: 23077 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves Jul 23 09:28:00 HA-xen001 ha-lizard: 23077 autoselect_slave: 1 available Slave UUIDs found: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:28:00 HA-xen001 ha-lizard: 23077 autoselect_slave: Selected Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 = Current slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 - ignoring update Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 check_xapi: Pool Host 192.168.1.61 xapi status = 1 Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 check_slave_status: Host IP Address check Status Array for Slaves = (1) Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 check_slave_status: Failed slave count = 0 Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 check_slave_status: No Failed slaves detected Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 check_ha_enabled: ha-lizard is enabled Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 Function check_slave_status reported no failures: calling vm_mon Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 vm_mon: Removing Control Domains from VM list Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running Jul 23 09:28:00 HA-xen001 ha-lizard: 23013 get_pool_ip_list: returned 192.168.1.61 192.168.1.60 Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running Jul 23 09:28:00 HA-xen001 ha-lizard: 23084 vm_mon: 0 Eligible Halted VMs found Jul 23 09:28:07 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:28:15 HA-xen001 ha-lizard: 23005 Spawning new instance of ha-lizard Jul 23 09:28:15 HA-xen001 ha-lizard: 23005 check_logger_processes: Checking logger processes Jul 23 09:28:15 HA-xen001 ha-lizard: 23005 check_logger_processes: No processes to clear Jul 23 09:28:15 HA-xen001 ha-lizard: 23837 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:28:15 HA-xen001 ha-lizard: 23837 This iteration is count 205 Jul 23 09:28:15 HA-xen001 ha-lizard: 23837 Checking if this host is a Pool Master or Slave Jul 23 09:28:15 HA-xen001 ha-lizard: 23837 This host's pool status = master Jul 23 09:28:15 HA-xen001 ha-lizard: 23837 Checking if ha-lizard is enabled for this pool Jul 23 09:28:17 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 check_ha_enabled: ha-lizard is enabled Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 ha-lizard is enabled Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 This host detected as pool Master Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 Found 2 hosts in pool Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 Calling function write_pool_state Jul 23 09:28:19 HA-xen001 ha-lizard: 23918 Calling function autoselect_slave Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 Calling function check_slave_status Jul 23 09:28:19 HA-xen001 ha-lizard: 23918 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:19 HA-xen001 ha-lizard: 23918 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 check_slave_status: Removing Master UUID from list of Hosts Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:28:19 HA-xen001 ha-lizard: 23918 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves Jul 23 09:28:19 HA-xen001 ha-lizard: 23918 autoselect_slave: 1 available Slave UUIDs found: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 Jul 23 09:28:19 HA-xen001 ha-lizard: 23918 autoselect_slave: Selected Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 = Current slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 - ignoring update Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 check_ha_enabled: ha-lizard is enabled Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 check_xapi: Pool Host 192.168.1.61 xapi status = 1 Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 check_slave_status: Host IP Address check Status Array for Slaves = (1) Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 check_slave_status: Failed slave count = 0 Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 check_slave_status: No Failed slaves detected Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 Function check_slave_status reported no failures: calling vm_mon Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 vm_mon: Removing Control Domains from VM list Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running Jul 23 09:28:19 HA-xen001 ha-lizard: 23837 get_pool_ip_list: returned 192.168.1.61 192.168.1.60 Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running Jul 23 09:28:19 HA-xen001 ha-lizard: 23925 vm_mon: 0 Eligible Halted VMs found Jul 23 09:28:27 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:28:30 HA-xen001 ha-lizard: 23829 Spawning new instance of ha-lizard Jul 23 09:28:30 HA-xen001 ha-lizard: 23829 check_logger_processes: Checking logger processes Jul 23 09:28:30 HA-xen001 ha-lizard: 23829 check_logger_processes: No processes to clear Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 This iteration is count 206 Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 Checking if this host is a Pool Master or Slave Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 This host's pool status = master Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 Checking if ha-lizard is enabled for this pool Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 check_ha_enabled: ha-lizard is enabled Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 ha-lizard is enabled Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 This host detected as pool Master Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 Found 2 hosts in pool Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 Calling function write_pool_state Jul 23 09:28:30 HA-xen001 ha-lizard: 24559 Calling function autoselect_slave Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 Calling function check_slave_status Jul 23 09:28:30 HA-xen001 ha-lizard: 24559 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:30 HA-xen001 ha-lizard: 24559 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 check_slave_status: Removing Master UUID from list of Hosts Jul 23 09:28:30 HA-xen001 ha-lizard: 24559 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves Jul 23 09:28:30 HA-xen001 ha-lizard: 24559 autoselect_slave: 1 available Slave UUIDs found: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:28:30 HA-xen001 ha-lizard: 24559 autoselect_slave: Selected Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 = Current slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 - ignoring update Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 check_xapi: Pool Host 192.168.1.61 xapi status = 1 Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 check_slave_status: Host IP Address check Status Array for Slaves = (1) Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 check_slave_status: Failed slave count = 0 Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 check_slave_status: No Failed slaves detected Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 Function check_slave_status reported no failures: calling vm_mon Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 check_ha_enabled: ha-lizard is enabled Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 vm_mon: Removing Control Domains from VM list Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running Jul 23 09:28:30 HA-xen001 ha-lizard: 24483 get_pool_ip_list: returned 192.168.1.61 192.168.1.60 Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running Jul 23 09:28:30 HA-xen001 ha-lizard: 24565 vm_mon: 0 Eligible Halted VMs found Jul 23 09:28:37 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:28:45 HA-xen001 ha-lizard: 24473 Spawning new instance of ha-lizard Jul 23 09:28:45 HA-xen001 ha-lizard: 24473 check_logger_processes: Checking logger processes Jul 23 09:28:45 HA-xen001 ha-lizard: 24473 check_logger_processes: No processes to clear Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 This iteration is count 207 Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 Checking if this host is a Pool Master or Slave Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 This host's pool status = master Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 Checking if ha-lizard is enabled for this pool Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 check_ha_enabled: ha-lizard is enabled Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 ha-lizard is enabled Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 This host detected as pool Master Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 Found 2 hosts in pool Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 Calling function write_pool_state Jul 23 09:28:45 HA-xen001 ha-lizard: 25397 Calling function autoselect_slave Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 Calling function check_slave_status Jul 23 09:28:45 HA-xen001 ha-lizard: 25397 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:45 HA-xen001 ha-lizard: 25397 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 check_slave_status: Removing Master UUID from list of Hosts Jul 23 09:28:45 HA-xen001 ha-lizard: 25397 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves Jul 23 09:28:45 HA-xen001 ha-lizard: 25397 autoselect_slave: 1 available Slave UUIDs found: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:28:45 HA-xen001 ha-lizard: 25397 autoselect_slave: Selected Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 = Current slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 - ignoring update Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 check_ha_enabled: ha-lizard is enabled Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 check_xapi: Pool Host 192.168.1.61 xapi status = 1 Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 check_slave_status: Host IP Address check Status Array for Slaves = (1) Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 check_slave_status: Failed slave count = 0 Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 check_slave_status: No Failed slaves detected Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 Function check_slave_status reported no failures: calling vm_mon Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Jul 23 09:28:45 HA-xen001 ha-lizard: 25333 get_pool_ip_list: returned 192.168.1.61 192.168.1.60 Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 vm_mon: Removing Control Domains from VM list Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running Jul 23 09:28:45 HA-xen001 ha-lizard: 25404 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running Jul 23 09:28:46 HA-xen001 ha-lizard: 25404 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running Jul 23 09:28:46 HA-xen001 ha-lizard: 25404 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running Jul 23 09:28:46 HA-xen001 ha-lizard: 25404 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running Jul 23 09:28:46 HA-xen001 ha-lizard: 25404 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running Jul 23 09:28:46 HA-xen001 ha-lizard: 25404 vm_mon: 0 Eligible Halted VMs found Jul 23 09:28:47 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:28:57 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:29:00 HA-xen001 ha-lizard: 25325 Spawning new instance of ha-lizard Jul 23 09:29:00 HA-xen001 ha-lizard: 25325 check_logger_processes: Checking logger processes Jul 23 09:29:00 HA-xen001 ha-lizard: 25325 check_logger_processes: No processes to clear Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 This iteration is count 208 Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 Checking if this host is a Pool Master or Slave Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 This host's pool status = master Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 Checking if ha-lizard is enabled for this pool Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 check_ha_enabled: ha-lizard is enabled Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 ha-lizard is enabled Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 This host detected as pool Master Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 Found 2 hosts in pool Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 Calling function write_pool_state Jul 23 09:29:00 HA-xen001 ha-lizard: 26084 Calling function autoselect_slave Jul 23 09:29:00 HA-xen001 ha-lizard: 26091 Calling function check_slave_status Jul 23 09:29:00 HA-xen001 ha-lizard: 26084 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:00 HA-xen001 ha-lizard: 26091 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:00 HA-xen001 ha-lizard: 26084 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:29:00 HA-xen001 ha-lizard: 26091 check_slave_status: Removing Master UUID from list of Hosts Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb Jul 23 09:29:00 HA-xen001 ha-lizard: 26084 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves Jul 23 09:29:00 HA-xen001 ha-lizard: 26084 autoselect_slave: 1 available Slave UUIDs found: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 Jul 23 09:29:00 HA-xen001 ha-lizard: 26091 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:29:00 HA-xen001 ha-lizard: 26084 autoselect_slave: Selected Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 = Current slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 - ignoring update Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 check_ha_enabled: ha-lizard is enabled Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:00 HA-xen001 ha-lizard: 26091 check_xapi: Pool Host 192.168.1.61 xapi status = 1 Jul 23 09:29:00 HA-xen001 ha-lizard: 26091 check_slave_status: Host IP Address check Status Array for Slaves = (1) Jul 23 09:29:00 HA-xen001 ha-lizard: 26091 check_slave_status: Failed slave count = 0 Jul 23 09:29:00 HA-xen001 ha-lizard: 26091 check_slave_status: No Failed slaves detected Jul 23 09:29:00 HA-xen001 ha-lizard: 26091 Function check_slave_status reported no failures: calling vm_mon Jul 23 09:29:00 HA-xen001 ha-lizard: 26091 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:29:00 HA-xen001 ha-lizard: 26091 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:29:00 HA-xen001 ha-lizard: 26091 vm_mon: Removing Control Domains from VM list Jul 23 09:29:00 HA-xen001 ha-lizard: 26091 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:29:00 HA-xen001 ha-lizard: 26016 get_pool_ip_list: returned 192.168.1.61 192.168.1.60 Jul 23 09:29:01 HA-xen001 ha-lizard: 26091 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running Jul 23 09:29:01 HA-xen001 ha-lizard: 26091 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running Jul 23 09:29:01 HA-xen001 ha-lizard: 26091 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running Jul 23 09:29:01 HA-xen001 ha-lizard: 26091 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running Jul 23 09:29:01 HA-xen001 ha-lizard: 26091 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running Jul 23 09:29:01 HA-xen001 ha-lizard: 26091 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running Jul 23 09:29:01 HA-xen001 ha-lizard: 26091 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running Jul 23 09:29:01 HA-xen001 ha-lizard: 26091 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running Jul 23 09:29:01 HA-xen001 ha-lizard: 26091 vm_mon: 0 Eligible Halted VMs found Jul 23 09:29:07 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:29:15 HA-xen001 ha-lizard: 26008 Spawning new instance of ha-lizard Jul 23 09:29:15 HA-xen001 ha-lizard: 26008 check_logger_processes: Checking logger processes Jul 23 09:29:15 HA-xen001 ha-lizard: 26008 check_logger_processes: No processes to clear Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 This iteration is count 209 Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 Checking if this host is a Pool Master or Slave Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 This host's pool status = master Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 Checking if ha-lizard is enabled for this pool Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 check_ha_enabled: ha-lizard is enabled Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 ha-lizard is enabled Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 This host detected as pool Master Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 Found 2 hosts in pool Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 Calling function write_pool_state Jul 23 09:29:15 HA-xen001 ha-lizard: 26967 Calling function autoselect_slave Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 Calling function check_slave_status Jul 23 09:29:15 HA-xen001 ha-lizard: 26967 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:15 HA-xen001 ha-lizard: 26967 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 check_slave_status: Removing Master UUID from list of Hosts Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:29:15 HA-xen001 ha-lizard: 26967 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves Jul 23 09:29:15 HA-xen001 ha-lizard: 26967 autoselect_slave: 1 available Slave UUIDs found: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 Jul 23 09:29:15 HA-xen001 ha-lizard: 26967 autoselect_slave: Selected Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 = Current slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 - ignoring update Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 check_xapi: Pool Host 192.168.1.61 xapi status = 1 Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 check_slave_status: Host IP Address check Status Array for Slaves = (1) Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 check_slave_status: Failed slave count = 0 Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 check_slave_status: No Failed slaves detected Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 Function check_slave_status reported no failures: calling vm_mon Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 check_ha_enabled: ha-lizard is enabled Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 vm_mon: Removing Control Domains from VM list Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running Jul 23 09:29:15 HA-xen001 ha-lizard: 26975 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running Jul 23 09:29:15 HA-xen001 ha-lizard: 26889 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:16 HA-xen001 ha-lizard: 26975 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running Jul 23 09:29:16 HA-xen001 ha-lizard: 26975 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running Jul 23 09:29:16 HA-xen001 ha-lizard: 26889 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:29:16 HA-xen001 ha-lizard: 26975 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running Jul 23 09:29:16 HA-xen001 ha-lizard: 26975 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running Jul 23 09:29:16 HA-xen001 ha-lizard: 26889 get_pool_ip_list: returned 192.168.1.61 192.168.1.60 Jul 23 09:29:16 HA-xen001 ha-lizard: 26975 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running Jul 23 09:29:16 HA-xen001 ha-lizard: 26975 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running Jul 23 09:29:16 HA-xen001 ha-lizard: 26975 vm_mon: 0 Eligible Halted VMs found Jul 23 09:29:17 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:29:27 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:29:30 HA-xen001 ha-lizard: 26879 Spawning new instance of ha-lizard Jul 23 09:29:30 HA-xen001 ha-lizard: 26879 check_logger_processes: Checking logger processes Jul 23 09:29:30 HA-xen001 ha-lizard: 26879 check_logger_processes: No processes to clear Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 This iteration is count 210 Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 Checking if this host is a Pool Master or Slave Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 This host's pool status = master Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 Checking if ha-lizard is enabled for this pool Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 check_ha_enabled: ha-lizard is enabled Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 ha-lizard is enabled Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 This host detected as pool Master Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 Found 2 hosts in pool Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 Calling function write_pool_state Jul 23 09:29:30 HA-xen001 ha-lizard: 28481 Calling function autoselect_slave Jul 23 09:29:30 HA-xen001 ha-lizard: 28486 Calling function check_slave_status Jul 23 09:29:30 HA-xen001 ha-lizard: 28481 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:30 HA-xen001 ha-lizard: 28486 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:30 HA-xen001 ha-lizard: 28481 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 Jul 23 09:29:30 HA-xen001 ha-lizard: 28486 check_slave_status: Removing Master UUID from list of Hosts Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:29:30 HA-xen001 ha-lizard: 28481 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves Jul 23 09:29:30 HA-xen001 ha-lizard: 28481 autoselect_slave: 1 available Slave UUIDs found: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 Jul 23 09:29:30 HA-xen001 ha-lizard: 28486 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:29:30 HA-xen001 ha-lizard: 28481 autoselect_slave: Selected Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 = Current slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 - ignoring update Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 check_ha_enabled: ha-lizard is enabled Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:30 HA-xen001 ha-lizard: 28486 check_xapi: Pool Host 192.168.1.61 xapi status = 1 Jul 23 09:29:30 HA-xen001 ha-lizard: 28486 check_slave_status: Host IP Address check Status Array for Slaves = (1) Jul 23 09:29:30 HA-xen001 ha-lizard: 28486 check_slave_status: Failed slave count = 0 Jul 23 09:29:30 HA-xen001 ha-lizard: 28486 check_slave_status: No Failed slaves detected Jul 23 09:29:30 HA-xen001 ha-lizard: 28486 Function check_slave_status reported no failures: calling vm_mon Jul 23 09:29:30 HA-xen001 ha-lizard: 28486 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Jul 23 09:29:30 HA-xen001 ha-lizard: 28407 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:29:31 HA-xen001 ha-lizard: 28486 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:29:31 HA-xen001 ha-lizard: 28486 vm_mon: Removing Control Domains from VM list Jul 23 09:29:31 HA-xen001 ha-lizard: 28486 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:29:31 HA-xen001 ha-lizard: 28407 get_pool_ip_list: returned 192.168.1.61 192.168.1.60 Jul 23 09:29:31 HA-xen001 ha-lizard: 28486 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running Jul 23 09:29:31 HA-xen001 ha-lizard: 28486 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running Jul 23 09:29:31 HA-xen001 ha-lizard: 28486 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running Jul 23 09:29:31 HA-xen001 ha-lizard: 28486 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running Jul 23 09:29:31 HA-xen001 ha-lizard: 28486 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running Jul 23 09:29:31 HA-xen001 ha-lizard: 28486 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running Jul 23 09:29:31 HA-xen001 ha-lizard: 28486 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running Jul 23 09:29:31 HA-xen001 ha-lizard: 28486 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running Jul 23 09:29:31 HA-xen001 ha-lizard: 28486 vm_mon: 0 Eligible Halted VMs found Jul 23 09:29:37 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:29:45 HA-xen001 ha-lizard: 28399 Spawning new instance of ha-lizard Jul 23 09:29:45 HA-xen001 ha-lizard: 28399 check_logger_processes: Checking logger processes Jul 23 09:29:45 HA-xen001 ha-lizard: 28399 check_logger_processes: No processes to clear Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 This iteration is count 211 Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 Checking if this host is a Pool Master or Slave Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 This host's pool status = master Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 Checking if ha-lizard is enabled for this pool Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 check_ha_enabled: ha-lizard is enabled Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 ha-lizard is enabled Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 This host detected as pool Master Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 Found 2 hosts in pool Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 Calling function write_pool_state Jul 23 09:29:45 HA-xen001 ha-lizard: 29306 Calling function autoselect_slave Jul 23 09:29:45 HA-xen001 ha-lizard: 29313 Calling function check_slave_status Jul 23 09:29:45 HA-xen001 ha-lizard: 29306 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:45 HA-xen001 ha-lizard: 29313 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:45 HA-xen001 ha-lizard: 29306 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:45 HA-xen001 ha-lizard: 29313 check_slave_status: Removing Master UUID from list of Hosts Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 Jul 23 09:29:45 HA-xen001 ha-lizard: 29306 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves Jul 23 09:29:45 HA-xen001 ha-lizard: 29306 autoselect_slave: 1 available Slave UUIDs found: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 Jul 23 09:29:45 HA-xen001 ha-lizard: 29313 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:29:45 HA-xen001 ha-lizard: 29306 autoselect_slave: Selected Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 = Current slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 - ignoring update Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 check_ha_enabled: ha-lizard is enabled Jul 23 09:29:45 HA-xen001 ha-lizard: 29236 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Jul 23 09:29:46 HA-xen001 ha-lizard: 29236 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 check_xapi: Pool Host 192.168.1.61 xapi status = 1 Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 check_slave_status: Host IP Address check Status Array for Slaves = (1) Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 check_slave_status: Failed slave count = 0 Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 check_slave_status: No Failed slaves detected Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 Function check_slave_status reported no failures: calling vm_mon Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Jul 23 09:29:46 HA-xen001 ha-lizard: 29236 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 vm_mon: Removing Control Domains from VM list Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:29:46 HA-xen001 ha-lizard: 29236 get_pool_ip_list: returned 192.168.1.61 192.168.1.60 Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running Jul 23 09:29:46 HA-xen001 ha-lizard: 29313 vm_mon: 0 Eligible Halted VMs found Jul 23 09:29:47 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:29:57 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:30:00 HA-xen001 ha-lizard: 29228 Spawning new instance of ha-lizard Jul 23 09:30:00 HA-xen001 ha-lizard: 29228 check_logger_processes: Checking logger processes Jul 23 09:30:00 HA-xen001 ha-lizard: 29228 check_logger_processes: No processes to clear Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 This iteration is count 212 Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 Checking if this host is a Pool Master or Slave Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 This host's pool status = master Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 Checking if ha-lizard is enabled for this pool Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 check_ha_enabled: ha-lizard is enabled Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 ha-lizard is enabled Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 This host detected as pool Master Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 Found 2 hosts in pool Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 Calling function write_pool_state Jul 23 09:30:00 HA-xen001 ha-lizard: 30749 Calling function autoselect_slave Jul 23 09:30:00 HA-xen001 ha-lizard: 30756 Calling function check_slave_status Jul 23 09:30:00 HA-xen001 ha-lizard: 30749 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:30:00 HA-xen001 ha-lizard: 30756 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:30:00 HA-xen001 ha-lizard: 30749 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:30:00 HA-xen001 ha-lizard: 30756 check_slave_status: Removing Master UUID from list of Hosts Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 Jul 23 09:30:00 HA-xen001 ha-lizard: 30756 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:30:00 HA-xen001 ha-lizard: 30749 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves Jul 23 09:30:00 HA-xen001 ha-lizard: 30749 autoselect_slave: 1 available Slave UUIDs found: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 Jul 23 09:30:00 HA-xen001 ha-lizard: 30675 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:30:00 HA-xen001 ha-lizard: 30749 autoselect_slave: Selected Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 = Current slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 - ignoring update Jul 23 09:30:00 HA-xen001 ha-lizard: 30756 check_xapi: Pool Host 192.168.1.61 xapi status = 1 Jul 23 09:30:00 HA-xen001 ha-lizard: 30756 check_slave_status: Host IP Address check Status Array for Slaves = (1) Jul 23 09:30:00 HA-xen001 ha-lizard: 30756 check_slave_status: Failed slave count = 0 Jul 23 09:30:00 HA-xen001 ha-lizard: 30756 check_slave_status: No Failed slaves detected Jul 23 09:30:00 HA-xen001 ha-lizard: 30756 Function check_slave_status reported no failures: calling vm_mon Jul 23 09:30:00 HA-xen001 ha-lizard: 30756 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Jul 23 09:30:01 HA-xen001 ha-lizard: 30756 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:30:01 HA-xen001 ha-lizard: 30756 vm_mon: Removing Control Domains from VM list Jul 23 09:30:01 HA-xen001 ha-lizard: 30756 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:30:01 HA-xen001 ha-lizard: 30675 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:30:01 HA-xen001 ha-lizard: 30675 check_ha_enabled: ha-lizard is enabled Jul 23 09:30:01 HA-xen001 ha-lizard: 30756 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running Jul 23 09:30:01 HA-xen001 ha-lizard: 30756 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running Jul 23 09:30:01 HA-xen001 ha-lizard: 30756 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running Jul 23 09:30:01 HA-xen001 ha-lizard: 30756 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running Jul 23 09:30:01 HA-xen001 ha-lizard: 30675 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Jul 23 09:30:01 HA-xen001 ha-lizard: 30756 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running Jul 23 09:30:01 HA-xen001 ha-lizard: 30756 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running Jul 23 09:30:01 HA-xen001 ha-lizard: 30675 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:30:01 HA-xen001 ha-lizard: 30756 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running Jul 23 09:30:01 HA-xen001 ha-lizard: 30756 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running Jul 23 09:30:01 HA-xen001 ha-lizard: 30756 vm_mon: 0 Eligible Halted VMs found Jul 23 09:30:01 HA-xen001 ha-lizard: 30675 get_pool_ip_list: returned 192.168.1.61 Jul 23 09:30:01 HA-xen001 ha-lizard: 30675 get_pool_ip_list: returned 192.168.1.61 192.168.1.60 Jul 23 09:30:07 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:30:15 HA-xen001 ha-lizard: 30665 Spawning new instance of ha-lizard Jul 23 09:30:15 HA-xen001 ha-lizard: 30665 check_logger_processes: Checking logger processes Jul 23 09:30:15 HA-xen001 ha-lizard: 30665 check_logger_processes: No processes to clear Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 This iteration is count 213 Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 Checking if this host is a Pool Master or Slave Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 This host's pool status = master Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 Checking if ha-lizard is enabled for this pool Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 check_ha_enabled: ha-lizard is enabled Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 ha-lizard is enabled Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 This host detected as pool Master Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 Found 2 hosts in pool Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 Calling function write_pool_state Jul 23 09:30:15 HA-xen001 ha-lizard: 31648 Calling function autoselect_slave Jul 23 09:30:15 HA-xen001 ha-lizard: 31652 Calling function check_slave_status Jul 23 09:30:15 HA-xen001 ha-lizard: 31648 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:30:15 HA-xen001 ha-lizard: 31652 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:30:15 HA-xen001 ha-lizard: 31648 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:30:15 HA-xen001 ha-lizard: 31576 get_vms_on_host: No VMs found on host: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 Jul 23 09:30:15 HA-xen001 ha-lizard: 31652 check_slave_status: Removing Slave UUID from list of Hosts - Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 is disabled or in maintenance mode Jul 23 09:30:15 HA-xen001 ha-lizard: 31652 check_slave_status: Removing Master UUID from list of Hosts Jul 23 09:30:15 HA-xen001 ha-lizard: 31652 check_slave_status: Host IP Address check Status Array for Slaves = () Jul 23 09:30:15 HA-xen001 ha-lizard: 31652 check_slave_status: Failed slave count = 0 Jul 23 09:30:15 HA-xen001 ha-lizard: 31652 check_slave_status: No Failed slaves detected Jul 23 09:30:15 HA-xen001 ha-lizard: 31652 Function check_slave_status reported no failures: calling vm_mon Jul 23 09:30:15 HA-xen001 ha-lizard: 31652 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Jul 23 09:30:16 HA-xen001 ha-lizard: 31576 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:30:16 HA-xen001 ha-lizard: 31648 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 is disabled or in maintenance mode Jul 23 09:30:16 HA-xen001 ha-lizard: 31648 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves Jul 23 09:30:16 HA-xen001 ha-lizard: 31648 autoselect_slave: 0 available Slave UUIDs found: Jul 23 09:30:16 HA-xen001 ha-lizard: 31652 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:30:16 HA-xen001 ha-lizard: 31652 vm_mon: Removing Control Domains from VM list Jul 23 09:30:16 HA-xen001 ha-lizard: 31652 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:30:16 HA-xen001 ha-lizard: 31648 autoselect_slave: New autopromote slave selected - updating pool-param:autopromote_uuid with Jul 23 09:30:16 HA-xen001 ha-lizard: 31648 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:30:16 HA-xen001 ha-lizard: 31648 email: Sending ALERT email to root@localhost: autoselect_slave: NOTICE: New autopromote slave selected - updating pool-param:autopromote_uuid with Jul 23 09:30:16 HA-xen001 ha-lizard: 31652 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running Jul 23 09:30:16 HA-xen001 ha-lizard: 31652 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running Jul 23 09:30:16 HA-xen001 ha-lizard: 31652 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running Jul 23 09:30:16 HA-xen001 ha-lizard: 31652 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running Jul 23 09:30:16 HA-xen001 ha-lizard: 31576 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:30:16 HA-xen001 ha-lizard: 31576 check_ha_enabled: ha-lizard is enabled Jul 23 09:30:16 HA-xen001 ha-lizard: 31652 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running Jul 23 09:30:16 HA-xen001 ha-lizard: 31652 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running Jul 23 09:30:16 HA-xen001 ha-lizard: 31652 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running Jul 23 09:30:16 HA-xen001 ha-lizard: 31652 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running Jul 23 09:30:16 HA-xen001 ha-lizard: 31652 vm_mon: 0 Eligible Halted VMs found Jul 23 09:30:16 HA-xen001 ha-lizard: 31576 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Jul 23 09:30:16 HA-xen001 ha-lizard: 31576 get_pool_host_list: returned a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:30:16 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email from: root@localhost Jul 23 09:30:16 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email to: root@localhost Jul 23 09:30:16 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:HA-xen001 Jul 23 09:30:16 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Jul 23 09:30:16 EDT 2014 Jul 23 09:30:16 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.7.7 Jul 23 09:30:16 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Content: autoselect_slave: NOTICE: New autopromote slave selected - updating pool-param:autopromote_uuid with Jul 23 09:30:16 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: HA-xen001 Jul 23 09:30:16 HA-xen001 ha-lizard: 31648 email: Message copied to /dev/shm/ha-lizard-mail/1406122216.msg - Suppressing duplicates for 60 Minutes Jul 23 09:30:16 HA-xen001 ha-lizard: 31576 get_pool_ip_list: returned 192.168.1.60 Jul 23 09:30:17 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:30:27 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK Jul 23 09:30:30 HA-xen001 ha-lizard: 31568 Spawning new instance of ha-lizard Jul 23 09:30:30 HA-xen001 ha-lizard: 31568 check_logger_processes: Checking logger processes Jul 23 09:30:30 HA-xen001 ha-lizard: 31568 check_logger_processes: No processes to clear Jul 23 09:30:30 HA-xen001 ha-lizard: 32151 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:30:30 HA-xen001 ha-lizard: 32151 This iteration is count 214 Jul 23 09:30:30 HA-xen001 ha-lizard: 32151 Checking if this host is a Pool Master or Slave Jul 23 09:30:30 HA-xen001 ha-lizard: 32151 This host's pool status = master Jul 23 09:30:30 HA-xen001 ha-lizard: 32151 Checking if ha-lizard is enabled for this pool Jul 23 09:30:30 HA-xen001 ha-lizard: 32151 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac Jul 23 09:30:30 HA-xen001 ha-lizard: 32151 check_ha_enabled: ha-lizard is enabled Jul 23 09:30:30 HA-xen001 ha-lizard: 32151 ha-lizard is enabled Jul 23 09:30:30 HA-xen001 ha-lizard: 32151 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Jul 23 09:30:30 HA-xen001 ha-lizard: 32151 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Jul 23 09:30:30 HA-xen001 ha-lizard: 32151 This host detected as pool Master Jul 23 09:30:30 HA-xen001 ha-lizard: 32151 Found 2 hosts in pool Jul 23 09:30:30 HA-xen001 ha-lizard: 32151 Calling function write_pool_state Jul 23 09:30:30 HA-xen001 ha-lizard: 32222 Calling function autoselect_slave Jul 23 09:30:30 HA-xen001 ha-lizard: 32230 Calling function check_slave_status Jul 23 09:30:30 HA-xen001 ha-lizard: 32222 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:30:30 HA-xen001 ha-lizard: 32230 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:30:30 HA-xen001 ha-lizard: 32222 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749 Jul 23 09:30:31 HA-xen001 ha-lizard: 32151 get_vms_on_host: No VMs found on host: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 check_slave_status: Removing Slave UUID from list of Hosts - Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 is disabled or in maintenance mode Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 check_slave_status: Removing Master UUID from list of Hosts Jul 23 09:30:31 HA-xen001 ha-lizard: 32222 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 is disabled or in maintenance mode Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 check_slave_status: Host IP Address check Status Array for Slaves = () Jul 23 09:30:31 HA-xen001 ha-lizard: 32222 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves Jul 23 09:30:31 HA-xen001 ha-lizard: 32222 autoselect_slave: 0 available Slave UUIDs found: Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 check_slave_status: Failed slave count = 0 Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 check_slave_status: No Failed slaves detected Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 Function check_slave_status reported no failures: calling vm_mon Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Jul 23 09:30:31 HA-xen001 ha-lizard: 32151 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 vm_mon: Removing Control Domains from VM list Jul 23 09:30:31 HA-xen001 ha-lizard: 32222 autoselect_slave: Selected Slave: = Current slave: - ignoring update Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4 Jul 23 09:30:31 HA-xen001 ha-lizard: 32151 Mail Spool Directory Found /dev/shm/ha-lizard-mail Jul 23 09:30:31 HA-xen001 ha-lizard: 32151 email: Sending ALERT email to root@localhost: write_pool_state: Error retrieving autopromote_uuid from pool configuration Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running Jul 23 09:30:31 HA-xen001 ha-lizard: 32230 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running Jul 23 09:30:31 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email from: root@localhost Jul 23 09:30:31 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email to: root@localhost Jul 23 09:30:31 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:HA-xen001 Jul 23 09:30:31 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Jul 23 09:30:31 EDT 2014 Jul 23 09:30:31 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.7.7 Jul 23 09:30:31 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Content: write_pool_state: Error retrieving autopromote_uuid from pool configuration Jul 23 09:30:31 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: HA-xen001 Jul 23 09:30:31 HA-xen001 ha-lizard: 32151 email: Message copied to /dev/shm/h