This will display a real time log session for ha-lizard
Control + C to exit.
Jul 23 11:14:23 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK 
Jul 23 11:14:24 HA-xen001 ha-lizard: 10250 Spawning new instance of ha-lizard
Jul 23 11:14:24 HA-xen001 ha-lizard: 10250 check_logger_processes: Checking logger processes
Jul 23 11:14:24 HA-xen001 ha-lizard: 10250 check_logger_processes: No processes to clear
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 This iteration is count 628
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 Checking if this host is a Pool Master or Slave
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 This host's pool status = master
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 Checking if ha-lizard is enabled for this pool
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 check_ha_enabled: ha-lizard is enabled
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 ha-lizard is enabled
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 This host detected as pool  Master
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 Found 2 hosts in pool
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 Calling function write_pool_state
Jul 23 11:14:24 HA-xen001 ha-lizard: 11138 Calling function autoselect_slave
Jul 23 11:14:24 HA-xen001 ha-lizard: 11145 Calling function check_slave_status
Jul 23 11:14:24 HA-xen001 ha-lizard: 11138 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:14:24 HA-xen001 ha-lizard: 11145 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:14:24 HA-xen001 ha-lizard: 11138 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4
Jul 23 11:14:24 HA-xen001 ha-lizard: 11145 check_slave_status: Removing Master UUID from list of Hosts
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb
Jul 23 11:14:24 HA-xen001 ha-lizard: 11138 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves
Jul 23 11:14:24 HA-xen001 ha-lizard: 11138 autoselect_slave: 1 available Slave UUIDs found: 3b3355ca-54ba-48ff-85a5-1c9724b39c15
Jul 23 11:14:24 HA-xen001 ha-lizard: 11145 get_pool_ip_list: returned 192.168.1.61
Jul 23 11:14:24 HA-xen001 ha-lizard: 11138 autoselect_slave: Selected Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 = Current slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 - ignoring update
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac
Jul 23 11:14:24 HA-xen001 ha-lizard: 11145 check_xapi: Pool Host 192.168.1.61 xapi status = 1
Jul 23 11:14:24 HA-xen001 ha-lizard: 11074 check_ha_enabled: ha-lizard is enabled
Jul 23 11:14:24 HA-xen001 ha-lizard: 11145 check_slave_status: Host IP Address check Status Array for Slaves = (1)
Jul 23 11:14:24 HA-xen001 ha-lizard: 11145 check_slave_status: Failed slave count = 0
Jul 23 11:14:24 HA-xen001 ha-lizard: 11145 check_slave_status: No Failed slaves detected
Jul 23 11:14:24 HA-xen001 ha-lizard: 11145 Function check_slave_status reported no failures: calling vm_mon
Jul 23 11:14:24 HA-xen001 ha-lizard: 11145 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Jul 23 11:14:25 HA-xen001 ha-lizard: 11145 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4
Jul 23 11:14:25 HA-xen001 ha-lizard: 11145 vm_mon: Removing Control Domains from VM list
Jul 23 11:14:25 HA-xen001 ha-lizard: 11145 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4
Jul 23 11:14:25 HA-xen001 ha-lizard: 11074 get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Jul 23 11:14:25 HA-xen001 ha-lizard: 11145 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running
Jul 23 11:14:25 HA-xen001 ha-lizard: 11145 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running
Jul 23 11:14:25 HA-xen001 ha-lizard: 11074 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:14:25 HA-xen001 ha-lizard: 11145 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running
Jul 23 11:14:25 HA-xen001 ha-lizard: 11145 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running
Jul 23 11:14:25 HA-xen001 ha-lizard: 11074 get_pool_ip_list: returned 192.168.1.61
Jul 23 11:14:25 HA-xen001 ha-lizard: 11145 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running
Jul 23 11:14:25 HA-xen001 ha-lizard: 11145 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running
Jul 23 11:14:25 HA-xen001 ha-lizard: 11074 get_pool_ip_list: returned 192.168.1.61 192.168.1.60
Jul 23 11:14:25 HA-xen001 ha-lizard: 11145 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running
Jul 23 11:14:25 HA-xen001 ha-lizard: 11145 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running
Jul 23 11:14:25 HA-xen001 ha-lizard: 11145 vm_mon: 0 Eligible Halted VMs found
Jul 23 11:14:33 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK 
Jul 23 11:14:39 HA-xen001 ha-lizard: 11066 Spawning new instance of ha-lizard
Jul 23 11:14:39 HA-xen001 ha-lizard: 11066 check_logger_processes: Checking logger processes
Jul 23 11:14:39 HA-xen001 ha-lizard: 11066 check_logger_processes: No processes to clear
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 This iteration is count 629
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 Checking if this host is a Pool Master or Slave
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 This host's pool status = master
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 Checking if ha-lizard is enabled for this pool
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 check_ha_enabled: ha-lizard is enabled
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 ha-lizard is enabled
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 This host detected as pool  Master
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 Found 2 hosts in pool
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 Calling function write_pool_state
Jul 23 11:14:39 HA-xen001 ha-lizard: 11747 Calling function autoselect_slave
Jul 23 11:14:39 HA-xen001 ha-lizard: 11754 Calling function check_slave_status
Jul 23 11:14:39 HA-xen001 ha-lizard: 11747 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:14:39 HA-xen001 ha-lizard: 11754 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:14:39 HA-xen001 ha-lizard: 11747 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4
Jul 23 11:14:39 HA-xen001 ha-lizard: 11754 check_slave_status: Removing Master UUID from list of Hosts
Jul 23 11:14:39 HA-xen001 ha-lizard: 11683 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb
Jul 23 11:14:39 HA-xen001 ha-lizard: 11754 get_pool_ip_list: returned 192.168.1.61
Jul 23 11:14:39 HA-xen001 ha-lizard: 11747 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves
Jul 23 11:14:39 HA-xen001 ha-lizard: 11747 autoselect_slave: 1 available Slave UUIDs found: 3b3355ca-54ba-48ff-85a5-1c9724b39c15
Jul 23 11:14:40 HA-xen001 ha-lizard: 11747 autoselect_slave: Selected Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 = Current slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 - ignoring update
Jul 23 11:14:40 HA-xen001 ha-lizard: 11683 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac
Jul 23 11:14:40 HA-xen001 ha-lizard: 11683 check_ha_enabled: ha-lizard is enabled
Jul 23 11:14:40 HA-xen001 ha-lizard: 11683 get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Jul 23 11:14:40 HA-xen001 ha-lizard: 11683 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:14:40 HA-xen001 ha-lizard: 11683 get_pool_ip_list: returned 192.168.1.61
Jul 23 11:14:40 HA-xen001 ha-lizard: 11683 get_pool_ip_list: returned 192.168.1.61 192.168.1.60
Jul 23 11:14:43 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK 
Jul 23 11:14:50 HA-xen001 ha-lizard: 11754 check_xapi: Pool Host 192.168.1.61 failed to respond to ICMP ping
Jul 23 11:14:50 HA-xen001 ha-lizard: 11754 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Jul 23 11:14:50 HA-xen001 ha-lizard: 11754 email: Sending ALERT email to root@localhost: check_xapi:  Pool Host on Server: 192.168.1.61 not responding to ICMP ping - manual intervention may be required
Jul 23 11:14:50 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email from: root@localhost
Jul 23 11:14:50 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email to: root@localhost
Jul 23 11:14:50 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:HA-xen001
Jul 23 11:14:50 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Jul 23 11:14:50 EDT 2014
Jul 23 11:14:50 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.7.7
Jul 23 11:14:50 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Content: check_xapi:  Pool Host on Server: 192.168.1.61 not responding to ICMP ping - manual intervention may be required
Jul 23 11:14:50 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: HA-xen001
Jul 23 11:14:50 HA-xen001 ha-lizard: 11754 email: Message copied to /dev/shm/ha-lizard-mail/1406128490.msg - Suppressing duplicates for 60 Minutes
Jul 23 11:14:50 HA-xen001 ha-lizard: 11754 check_slave_status: Attempt 0: Checking Slave Status
Jul 23 11:14:53 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK 
Jul 23 11:14:54 HA-xen001 ha-lizard: 11675 ha-lizard already running: Attempt 1 on PIDS: 11683 11678 11677 11675
Jul 23 11:15:03 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK 
Jul 23 11:15:04 HA-xen001 ha-lizard: 11675 ha-lizard already running: Attempt 2 on PIDS: 11683 11678 11677 11675
Jul 23 11:15:10 HA-xen001 ha-lizard: 11754 check_xapi: Pool Host 192.168.1.61 failed to respond to ICMP ping
Jul 23 11:15:10 HA-xen001 ha-lizard: 11754 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Jul 23 11:15:10 HA-xen001 ha-lizard: 11754 email: Duplicate message - not sending. Content = check_xapi:  Pool Host on Server: 192.168.1.61 not responding to ICMP ping - manual intervention may be required
Jul 23 11:15:10 HA-xen001 ha-lizard: 11754 email: Message barred for 60 minutes
Jul 23 11:15:10 HA-xen001 ha-lizard: 11754 check_slave_status: Attempt 1: Checking Slave Status
Jul 23 11:15:13 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK 
Jul 23 11:15:14 HA-xen001 ha-lizard: 11675 ha-lizard already running: Attempt 3 on PIDS: 11683 11678 11677 11675
Jul 23 11:15:23 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK 
Jul 23 11:15:24 HA-xen001 ha-lizard: 11675 ha-lizard already running: Attempt 4 on PIDS: 11683 11678 11677 11675
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 check_xapi: Pool Host 192.168.1.61 failed to respond to ICMP ping
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 email: Duplicate message - not sending. Content = check_xapi:  Pool Host on Server: 192.168.1.61 not responding to ICMP ping - manual intervention may be required
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 email: Message barred for 60 minutes
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 check_slave_status: Failed to reach host
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 check_slave_status: Host IP Address check Status Array for Slaves = (0)
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 check_slave_status: Failed slave count = 1
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 check_slave_status: Processing failed slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 on this iteration
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 email: Sending ALERT email to root@localhost: check_slave_status:  Server HA-xen001: Some Pool Slaves not not responding , 3b3355ca-54ba-48ff-85a5-1c9724b39c15
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email from: root@localhost
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email to: root@localhost
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:HA-xen001
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Jul 23 11:15:30 EDT 2014
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.7.7
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Content: check_slave_status:  Server HA-xen001: Some Pool Slaves not not responding , 3b3355ca-54ba-48ff-85a5-1c9724b39c15
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: HA-xen001
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 email: Message copied to /dev/shm/ha-lizard-mail/1406128530.msg - Suppressing duplicates for 60 Minutes
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 check_slave_status: Some Pool Slaves not not responding , 3b3355ca-54ba-48ff-85a5-1c9724b39c15
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 check_slave_status: Calling function get_vms_on_host for UUID(s) 3b3355ca-54ba-48ff-85a5-1c9724b39c15
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 check_slave_status: Calling function fence_host to remove unresponsive host from pool. Failed Host(s) = 3b3355ca-54ba-48ff-85a5-1c9724b39c15
Jul 23 11:15:30 HA-xen001 ha-lizard: 11754 check_slave_status: fence_host host = 3b3355ca-54ba-48ff-85a5-1c9724b39c15, action = stop
Jul 23 11:15:30 HA-xen001 ha-lizard: 12967 fence_host: fence_host called to fencehost:  3b3355ca-54ba-48ff-85a5-1c9724b39c15 with action: stop and method: POOL. Check Pool!
Jul 23 11:15:30 HA-xen001 ha-lizard: 12967 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Jul 23 11:15:30 HA-xen001 ha-lizard: 12967 email: Sending ALERT email to root@localhost: fence_host:  fence_host called to fencehost:  3b3355ca-54ba-48ff-85a5-1c9724b39c15 with action: stop and method: POOL. Check Pool!
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email from: root@localhost
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email to: root@localhost
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:HA-xen001
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Jul 23 11:15:30 EDT 2014
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.7.7
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Content: fence_host:  fence_host called to fencehost:  3b3355ca-54ba-48ff-85a5-1c9724b39c15 with action: stop and method: POOL. Check Pool!
Jul 23 11:15:30 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: HA-xen001
Jul 23 11:15:30 HA-xen001 ha-lizard: 12967 email: Message copied to /dev/shm/ha-lizard-mail/1406128530.msg - Suppressing duplicates for 60 Minutes
Jul 23 11:15:30 HA-xen001 ha-lizard: 12967 check_quorum: Checking host IPs: 192.168.1.61 192.168.1.60
Jul 23 11:15:33 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK 
Jul 23 11:15:33 HA-xen001 ha-lizard: 12967 check_quorum: HOST IP: 192.168.1.61 Response = Fail
Jul 23 11:15:33 HA-xen001 ha-lizard: 12967 check_quorum: Host IP: 192.168.1.60 Response = OK
Jul 23 11:15:33 HA-xen001 ha-lizard: 12967 check_quorum: LIVE HOSTs = 1
Jul 23 11:15:33 HA-xen001 ha-lizard: 12967 check_quorum: Using network points: 192.168.1.253 as possible additional vote
Jul 23 11:15:34 HA-xen001 ha-lizard: 11675 ha-lizard already running: Attempt 5 on PIDS: 12967 12966 11683 11678 11677 11675
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 check_quorum: Heuristic IP: 192.168.1.253 Response = Fail
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 Total enpoints checked = 1 with total successful replies = 0
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 check_quorum: Failed to add additional vote by checking heuristics
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 email: Sending ALERT email to root@localhost: check_quorum:  Failed to add additional vote by checking heuristics
Jul 23 11:15:36 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email from: root@localhost
Jul 23 11:15:36 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email to: root@localhost
Jul 23 11:15:36 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:HA-xen001
Jul 23 11:15:36 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Jul 23 11:15:36 EDT 2014
Jul 23 11:15:36 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.7.7
Jul 23 11:15:36 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Content: check_quorum:  Failed to add additional vote by checking heuristics
Jul 23 11:15:36 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: HA-xen001
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 email: Message copied to /dev/shm/ha-lizard-mail/1406128536.msg - Suppressing duplicates for 60 Minutes
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 check_quorum: 1 Hosts found. Minimum needed 1 + 1. Fencing blocked
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 fence_host: Remaining hosts do not have quorum - setting FENCE_BLOCK
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 fence_host: Fencing is disabled or blocked. Check configuration or pool state. FENCE_ENABLED = 1, FENCE_BLOCK = 1, NUM_HOSTS = 2, FENCE_MIN_HOSTS = 2 - cannot fence host 3b3355ca-54ba-48ff-85a5-1c9724b39c15
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 fence_host: Resetting API..
Jul 23 11:15:39 HA-xen001 ha-lizard-ERROR-/etc/ha-lizard/ha-lizard.sh: /etc/ha-lizard/ha-lizard.func: line 973: Stopping: command not found
Jul 23 11:15:43 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK 
Jul 23 11:15:44 HA-xen001 ha-lizard: 11675 ha-lizard already running: Attempt 6 on PIDS: 12967 12966 11683 11678 11677 11675
Jul 23 11:15:49 HA-xen001 ha-lizard: 12967 check_slave_status: fence_host return status = 1
Jul 23 11:15:49 HA-xen001 ha-lizard: 12967 check_slave_status: fence_host failed to fence failed host: 3b3355ca-54ba-48ff-85a5-1c9724b39c15
Jul 23 11:15:49 HA-xen001 ha-lizard: 12967 check_slave_status: Host: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 power state unknown
Jul 23 11:15:49 HA-xen001 ha-lizard: 12967 Function check_slave_status failed to fence failed host.. checking whether to attempt starting failed VMs
Jul 23 11:15:49 HA-xen001 ha-lizard: 12967 FENCE_HA_ONFAIL is set to: 0
Jul 23 11:15:49 HA-xen001 ha-lizard: 12967 FENCE_HA_ONFAIL is set to: 0, not attempting to start VMs
Jul 23 11:15:53 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK 
Jul 23 11:15:54 HA-xen001 ha-lizard: 11675 Spawning new instance of ha-lizard
Jul 23 11:15:54 HA-xen001 ha-lizard: 11675 check_logger_processes: Checking logger processes
Jul 23 11:15:54 HA-xen001 ha-lizard: 11675 check_logger_processes: No processes to clear
Jul 23 11:15:54 HA-xen001 ha-lizard: 14343 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Jul 23 11:15:54 HA-xen001 ha-lizard: 14343 This iteration is count 630
Jul 23 11:15:54 HA-xen001 ha-lizard: 14343 Checking if this host is a Pool Master or Slave
Jul 23 11:15:54 HA-xen001 ha-lizard: 14343 This host's pool status = master
Jul 23 11:15:54 HA-xen001 ha-lizard: 14343 Checking if ha-lizard is enabled for this pool
Jul 23 11:15:54 HA-xen001 ha-lizard: 14343 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac
Jul 23 11:15:54 HA-xen001 ha-lizard: 14343 check_ha_enabled: ha-lizard is enabled
Jul 23 11:15:54 HA-xen001 ha-lizard: 14343 ha-lizard is enabled
Jul 23 11:15:55 HA-xen001 ha-lizard: 14343 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run
Jul 23 11:15:55 HA-xen001 ha-lizard: 14343 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10
Jul 23 11:15:55 HA-xen001 ha-lizard: 14343 This host detected as pool  Master
Jul 23 11:15:55 HA-xen001 ha-lizard: 14343 Found 2 hosts in pool
Jul 23 11:15:55 HA-xen001 ha-lizard: 14343 Calling function write_pool_state
Jul 23 11:15:55 HA-xen001 ha-lizard: 14407 Calling function autoselect_slave
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 Calling function check_slave_status
Jul 23 11:15:55 HA-xen001 ha-lizard: 14407 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:15:55 HA-xen001 ha-lizard: 14407 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:15:55 HA-xen001 ha-lizard: 14343 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 check_slave_status: Removing Slave UUID from list of Hosts - Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 is disabled or in maintenance mode
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 check_slave_status: Removing Master UUID from list of Hosts
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 check_slave_status: Host IP Address check Status Array for Slaves = ()
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 check_slave_status: Failed slave count = 0
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 check_slave_status: No Failed slaves detected
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 Function check_slave_status reported no failures: calling vm_mon
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Jul 23 11:15:55 HA-xen001 ha-lizard: 14343 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 vm_mon: Removing Control Domains from VM list
Jul 23 11:15:55 HA-xen001 ha-lizard: 14407 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 is disabled or in maintenance mode
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4
Jul 23 11:15:55 HA-xen001 ha-lizard: 14407 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves
Jul 23 11:15:55 HA-xen001 ha-lizard: 14407 autoselect_slave: 0 available Slave UUIDs found: 
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running
Jul 23 11:15:55 HA-xen001 ha-lizard: 14407 autoselect_slave: New autopromote slave selected - updating pool-param:autopromote_uuid with 
Jul 23 11:15:55 HA-xen001 ha-lizard: 14407 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Jul 23 11:15:55 HA-xen001 ha-lizard: 14407 email: Sending ALERT email to root@localhost: autoselect_slave:  NOTICE: New autopromote slave selected - updating pool-param:autopromote_uuid with 
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running
Jul 23 11:15:55 HA-xen001 ha-lizard: 14343 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running
Jul 23 11:15:55 HA-xen001 ha-lizard: 14343 check_ha_enabled: ha-lizard is enabled
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 vm_mon: VM 609d846d-8692-3383-3877-75a25a5c39a4 state = running
Jul 23 11:15:55 HA-xen001 ha-lizard: 14414 vm_mon: 0 Eligible Halted VMs found
Jul 23 11:15:55 HA-xen001 ha-lizard: 14343 get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Jul 23 11:15:55 HA-xen001 ha-lizard: 14343 get_pool_host_list: returned a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:15:55 HA-xen001 ha-lizard: 14343 get_pool_ip_list: returned 192.168.1.60
Jul 23 11:15:55 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email from: root@localhost
Jul 23 11:15:55 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email to: root@localhost
Jul 23 11:15:55 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:HA-xen001
Jul 23 11:15:55 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Jul 23 11:15:55 EDT 2014
Jul 23 11:15:55 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.7.7
Jul 23 11:15:55 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Content: autoselect_slave:  NOTICE: New autopromote slave selected - updating pool-param:autopromote_uuid with 
Jul 23 11:15:55 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: HA-xen001
Jul 23 11:15:55 HA-xen001 ha-lizard: 14407 email: Message copied to /dev/shm/ha-lizard-mail/1406128555.msg - Suppressing duplicates for 60 Minutes
Jul 23 11:16:03 HA-xen001 ha-lizard: 6945 ha-lizard Watchdog: ha-lizard running - OK 
Jul 23 11:16:09 HA-xen001 ha-lizard: 14335 Spawning new instance of ha-lizard
Jul 23 11:16:09 HA-xen001 ha-lizard: 14335 check_logger_processes: Checking logger processes
Jul 23 11:16:09 HA-xen001 ha-lizard: 14335 check_logger_processes: No processes to clear
Jul 23 11:16:09 HA-xen001 ha-lizard: 14994 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Jul 23 11:16:09 HA-xen001 ha-lizard: 14994 This iteration is count 631
Jul 23 11:16:09 HA-xen001 ha-lizard: 14994 Checking if this host is a Pool Master or Slave
Jul 23 11:16:09 HA-xen001 ha-lizard: 14994 This host's pool status = master
Jul 23 11:16:09 HA-xen001 ha-lizard: 14994 Checking if ha-lizard is enabled for this pool
Jul 23 11:16:10 HA-xen001 ha-lizard: 14994 check_ha_enabled: Checking if ha-lizard is enabled for pool: 54a2e454-2417-22fd-362f-4a17189168ac
Jul 23 11:16:10 HA-xen001 ha-lizard: 14994 check_ha_enabled: ha-lizard is enabled
Jul 23 11:16:10 HA-xen001 ha-lizard: 14994 ha-lizard is enabled
Jul 23 11:16:10 HA-xen001 ha-lizard: 14994 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run
Jul 23 11:16:10 HA-xen001 ha-lizard: 14994 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10
Jul 23 11:16:10 HA-xen001 ha-lizard: 14994 This host detected as pool  Master
Jul 23 11:16:10 HA-xen001 ha-lizard: 14994 Found 2 hosts in pool
Jul 23 11:16:10 HA-xen001 ha-lizard: 14994 Calling function write_pool_state
Jul 23 11:16:10 HA-xen001 ha-lizard: 15058 Calling function autoselect_slave
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 Calling function check_slave_status
Jul 23 11:16:10 HA-xen001 ha-lizard: 15058 autoselect_slave: This host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 get_pool_host_list: returned 3b3355ca-54ba-48ff-85a5-1c9724b39c15 a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:16:10 HA-xen001 ha-lizard: 15058 autoselect_slave: MASTER host UUID found: a4ba9a07-d9ea-494a-b529-f00df037e749
Jul 23 11:16:10 HA-xen001 ha-lizard: 14994 get_vms_on_host: Returned f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 check_slave_status: Removing Slave UUID from list of Hosts - Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 is disabled or in maintenance mode
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 check_slave_status: Removing Master UUID from list of Hosts
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 check_slave_status: Host IP Address check Status Array for Slaves = ()
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 check_slave_status: Failed slave count = 0
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 check_slave_status: No Failed slaves detected
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 Function check_slave_status reported no failures: calling vm_mon
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Jul 23 11:16:10 HA-xen001 ha-lizard: 14994 get_vms_on_host: Returned 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb
Jul 23 11:16:10 HA-xen001 ha-lizard: 15058 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 is disabled or in maintenance mode
Jul 23 11:16:10 HA-xen001 ha-lizard: 15058 autoselect_slave: a4ba9a07-d9ea-494a-b529-f00df037e749 is Master UUID - excluding from list of available slaves
Jul 23 11:16:10 HA-xen001 ha-lizard: 15058 autoselect_slave: 0 available Slave UUIDs found: 
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 vm_mon: Retrived list of VMs for this poll: 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 vm_mon: Removing Control Domains from VM list
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 vm_mon: VM list returned = 838ee264-5d94-b783-fe1e-cdfbd598daee 4abc113e-9e02-b8d3-b977-9dd60a7345fb f80abd72-c0be-5faa-7663-5e8fc02606a7 609d846d-8692-3383-3877-75a25a5c39a4
Jul 23 11:16:10 HA-xen001 ha-lizard: 14994 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Jul 23 11:16:10 HA-xen001 ha-lizard: 15058 autoselect_slave: Selected Slave:  = Current slave:  - ignoring update
Jul 23 11:16:10 HA-xen001 ha-lizard: 14994 email: Sending ALERT email to root@localhost: write_pool_state:  Error retrieving autopromote_uuid from pool configuration
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 vm_state: Machine state for 838ee264-5d94-b783-fe1e-cdfbd598daee returned: running
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 vm_mon: VM 838ee264-5d94-b783-fe1e-cdfbd598daee state = running
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 vm_state: Machine state for 4abc113e-9e02-b8d3-b977-9dd60a7345fb returned: running
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 vm_mon: VM 4abc113e-9e02-b8d3-b977-9dd60a7345fb state = running
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 vm_state: Machine state for f80abd72-c0be-5faa-7663-5e8fc02606a7 returned: running
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 vm_mon: VM f80abd72-c0be-5faa-7663-5e8fc02606a7 state = running
Jul 23 11:16:10 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email from: root@localhost
Jul 23 11:16:10 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Sending email to: root@localhost
Jul 23 11:16:10 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:HA-xen001
Jul 23 11:16:10 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Jul 23 11:16:10 EDT 2014
Jul 23 11:16:10 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.7.7
Jul 23 11:16:10 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Content: write_pool_state:  Error retrieving autopromote_uuid from pool configuration
Jul 23 11:16:10 HA-xen001 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: HA-xen001
Jul 23 11:16:10 HA-xen001 ha-lizard: 14994 email: Message copied to /dev/shm/ha-lizard-mail/1406128570.msg - Suppressing duplicates for 60 Minutes
Jul 23 11:16:10 HA-xen001 ha-lizard: 15065 vm_state: Machine state for 609d846d-8692-3383-3877-75a25a5c39a4 returned: running
Jul 23 11:16:10 HA-xen001 h