[root@xs2 ~]# tail -f /var/log/messages | grep -v email | grep -v xapi Sep 9 22:58:56 xs2 iscsi-ha: 17619 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 22:58:56 xs2 iscsi-ha: 17619 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- ns:0 nr:12292 dw:12292 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 22:58:56 xs2 iscsi-ha: 17619 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 22:58:56 xs2 iscsi-ha: 17619 DRBD Resource: iscsi1 in Connected state Sep 9 22:58:56 xs2 iscsi-ha-17575-INFO-/etc/iscsi-ha/iscsi-ha.sh: Scanning for Volume Group -> iscsi-sr: 4a13ab89-f5b5-e2e4-978b-73127607eae4 Sep 9 22:58:56 xs2 iscsi-ha-17575-INFO-/etc/iscsi-ha/iscsi-ha.sh: Volume Group for iSCSI-SR found OK: 4a13ab89-f5b5-e2e4-978b-73127607eae4 Sep 9 22:59:02 xs2 ha-lizard: 3936 ha-lizard Watchdog: ha-lizard running - OK Sep 9 22:59:03 xs2 iscsi-ha: 4150 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 22:59:06 xs2 iscsi-ha: 17570 Spawning new instance of iscsi-ha Sep 9 22:59:06 xs2 iscsi-ha: 17570 check_logger_processes Checking logger processes Sep 9 22:59:06 xs2 iscsi-ha: 17570 check_logger_processes No processes to clear Sep 9 22:59:06 xs2 iscsi-ha: 17767 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 22:59:06 xs2 iscsi-ha: 17767 This iteration is count 29 Sep 9 22:59:06 xs2 iscsi-ha: 17767 Checking if this host is a Pool Master or Slave Sep 9 22:59:06 xs2 iscsi-ha: 17767 This host's pool status = slave:192.168.199.200 Sep 9 22:59:06 xs2 iscsi-ha: 17801 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 22:59:06 xs2 iscsi-ha: 17801 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 22:59:06 xs2 iscsi-ha: 17801 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- ns:0 nr:12292 dw:12292 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 22:59:06 xs2 iscsi-ha: 17801 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 22:59:06 xs2 iscsi-ha: 17801 DRBD Resource: iscsi1 in Connected state Sep 9 22:59:06 xs2 iscsi-ha-17762-INFO-/etc/iscsi-ha/iscsi-ha.sh: Scanning for Volume Group -> iscsi-sr: 4a13ab89-f5b5-e2e4-978b-73127607eae4 Sep 9 22:59:06 xs2 iscsi-ha-17762-INFO-/etc/iscsi-ha/iscsi-ha.sh: Volume Group for iSCSI-SR found OK: 4a13ab89-f5b5-e2e4-978b-73127607eae4 Sep 9 22:59:08 xs2 ha-lizard: 17271 Spawning new instance of ha-lizard Sep 9 22:59:08 xs2 ha-lizard: 17871 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 22:59:08 xs2 ha-lizard: 17871 This iteration is count 20 Sep 9 22:59:08 xs2 ha-lizard: 17871 Checking if this host is a Pool Master or Slave Sep 9 22:59:08 xs2 ha-lizard: 17871 This host's pool status = slave:192.168.199.200 Sep 9 22:59:09 xs2 ha-lizard: 17871 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Sep 9 22:59:09 xs2 ha-lizard: 17871 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.199.1 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 SMTP_PASS="" SMTP_PORT="25" SMTP_SERVER="127.0.0.1" SMTP_USER="" XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Sep 9 22:59:09 xs2 ha-lizard: 17871 master_ip: Pool Master IP Address = 192.168.199.200 Sep 9 22:59:09 xs2 ha-lizard: 17871 This slave - xs2: 79769bcb-ad49-4ce1-8b27-51234c5b418f selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1 Sep 9 22:59:09 xs2 ha-lizard: 17871 Pool Master is OK - calling function check_ha_enabled - updating local status Sep 9 22:59:09 xs2 ha-lizard: 17871 check_ha_enabled: Checking if ha-lizard is enabled for pool: 3243bef3-b686-d02f-0d53-9e9529ee9639 Sep 9 22:59:09 xs2 ha-lizard: 17871 check_ha_enabled: ha-lizard is enabled Sep 9 22:59:09 xs2 ha-lizard: 17871 Checking state file for status if ha-lizard is enabled Sep 9 22:59:09 xs2 ha-lizard: 17871 Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled Sep 9 22:59:09 xs2 ha-lizard: 17871 ha-lizard is enabled - continuing Sep 9 22:59:09 xs2 ha-lizard: 17871 check_xs_ha: Checking XenServer HA status Sep 9 22:59:09 xs2 ha-lizard: 17871 Calling Function write_pool_state - updating local state files Sep 9 22:59:10 xs2 ha-lizard: 17871 get_vms_on_host: No VMs found on host: 26c15cda-7a26-41e8-a2f7-cdd7a9d5df8f Sep 9 22:59:10 xs2 ha-lizard: 17871 get_vms_on_host: No VMs found on host: 79769bcb-ad49-4ce1-8b27-51234c5b418f Sep 9 22:59:10 xs2 ha-lizard: 17871 check_ha_enabled: Checking if ha-lizard is enabled for pool: 3243bef3-b686-d02f-0d53-9e9529ee9639 Sep 9 22:59:10 xs2 ha-lizard: 17871 check_ha_enabled: ha-lizard is enabled Sep 9 22:59:11 xs2 ha-lizard: 17871 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Sep 9 22:59:11 xs2 ha-lizard: 17871 get_pool_host_list: returned 26c15cda-7a26-41e8-a2f7-cdd7a9d5df8f 79769bcb-ad49-4ce1-8b27-51234c5b418f Sep 9 22:59:11 xs2 ha-lizard: 17871 get_pool_ip_list: returned 192.168.199.200 Sep 9 22:59:11 xs2 ha-lizard: 17871 get_pool_ip_list: returned 192.168.199.200 192.168.199.159 Sep 9 22:59:12 xs2 ha-lizard: 3936 ha-lizard Watchdog: ha-lizard running - OK Sep 9 22:59:13 xs2 iscsi-ha: 4150 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 22:59:16 xs2 iscsi-ha: 17759 Spawning new instance of iscsi-ha Sep 9 22:59:16 xs2 iscsi-ha: 17759 check_logger_processes Checking logger processes Sep 9 22:59:16 xs2 iscsi-ha: 17759 check_logger_processes No processes to clear Sep 9 22:59:16 xs2 iscsi-ha: 18211 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 22:59:16 xs2 iscsi-ha: 18211 This iteration is count 30 Sep 9 22:59:16 xs2 iscsi-ha: 18211 Checking if this host is a Pool Master or Slave Sep 9 22:59:16 xs2 iscsi-ha: 18211 This host's pool status = slave:192.168.199.200 Sep 9 22:59:16 xs2 iscsi-ha: 18256 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 22:59:16 xs2 iscsi-ha: 18256 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 22:59:16 xs2 iscsi-ha: 18256 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- ns:0 nr:12292 dw:12292 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 22:59:16 xs2 iscsi-ha: 18256 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 22:59:16 xs2 iscsi-ha: 18256 DRBD Resource: iscsi1 in Connected state Sep 9 22:59:17 xs2 iscsi-ha-18206-INFO-/etc/iscsi-ha/iscsi-ha.sh: Scanning for Volume Group -> iscsi-sr: 4a13ab89-f5b5-e2e4-978b-73127607eae4 Sep 9 22:59:17 xs2 iscsi-ha-18206-INFO-/etc/iscsi-ha/iscsi-ha.sh: Volume Group for iSCSI-SR found OK: 4a13ab89-f5b5-e2e4-978b-73127607eae4 Sep 9 22:59:22 xs2 ha-lizard: 3936 ha-lizard Watchdog: ha-lizard running - OK Sep 9 22:59:23 xs2 iscsi-ha: 4150 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 22:59:23 xs2 ha-lizard: 17863 Spawning new instance of ha-lizard Sep 9 22:59:23 xs2 ha-lizard: 18357 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 22:59:23 xs2 ha-lizard: 18357 This iteration is count 21 Sep 9 22:59:23 xs2 ha-lizard: 18357 Checking if this host is a Pool Master or Slave Sep 9 22:59:23 xs2 ha-lizard: 18357 This host's pool status = slave:192.168.199.200 Sep 9 22:59:24 xs2 ha-lizard: 18357 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run Sep 9 22:59:24 xs2 ha-lizard: 18357 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=1 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=0 FENCE_HEURISTICS_IPS=192.168.199.1 FENCE_HOST_FORGET=0 FENCE_IPADDRESS= FENCE_METHOD=POOL FENCE_MIN_HOSTS=2 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=1 MAIL_FROM="root@localhost" MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO="root@localhost" MONITOR_DELAY=15 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=20 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 SMTP_PASS="" SMTP_PORT="25" SMTP_SERVER="127.0.0.1" SMTP_USER="" XAPI_COUNT=2 XAPI_DELAY=10 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10 Sep 9 22:59:24 xs2 ha-lizard: 18357 master_ip: Pool Master IP Address = 192.168.199.200 Sep 9 22:59:24 xs2 ha-lizard: 18357 This slave - xs2: 79769bcb-ad49-4ce1-8b27-51234c5b418f selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1 Sep 9 22:59:24 xs2 ha-lizard: 18357 Pool Master is OK - calling function check_ha_enabled - updating local status Sep 9 22:59:24 xs2 ha-lizard: 18357 check_ha_enabled: Checking if ha-lizard is enabled for pool: 3243bef3-b686-d02f-0d53-9e9529ee9639 Sep 9 22:59:24 xs2 ha-lizard: 18357 check_ha_enabled: ha-lizard is enabled Sep 9 22:59:24 xs2 ha-lizard: 18357 Checking state file for status if ha-lizard is enabled Sep 9 22:59:24 xs2 ha-lizard: 18357 Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled Sep 9 22:59:24 xs2 ha-lizard: 18357 ha-lizard is enabled - continuing Sep 9 22:59:24 xs2 ha-lizard: 18357 check_xs_ha: Checking XenServer HA status Sep 9 22:59:24 xs2 ha-lizard: 18357 Calling Function write_pool_state - updating local state files Sep 9 22:59:25 xs2 ha-lizard: 18357 get_vms_on_host: No VMs found on host: 26c15cda-7a26-41e8-a2f7-cdd7a9d5df8f Sep 9 22:59:25 xs2 ha-lizard: 18357 get_vms_on_host: No VMs found on host: 79769bcb-ad49-4ce1-8b27-51234c5b418f Sep 9 22:59:25 xs2 ha-lizard: 18357 check_ha_enabled: Checking if ha-lizard is enabled for pool: 3243bef3-b686-d02f-0d53-9e9529ee9639 Sep 9 22:59:25 xs2 ha-lizard: 18357 check_ha_enabled: ha-lizard is enabled Sep 9 22:59:26 xs2 ha-lizard: 18357 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Sep 9 22:59:26 xs2 ha-lizard: 18357 get_pool_host_list: returned 79769bcb-ad49-4ce1-8b27-51234c5b418f Sep 9 22:59:26 xs2 ha-lizard: 18357 get_pool_ip_list: returned 192.168.199.159 Sep 9 22:59:26 xs2 iscsi-ha: 18203 Spawning new instance of iscsi-ha Sep 9 22:59:26 xs2 iscsi-ha: 18203 check_logger_processes Checking logger processes Sep 9 22:59:26 xs2 iscsi-ha: 18203 check_logger_processes No processes to clear Sep 9 22:59:26 xs2 iscsi-ha: 18651 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 22:59:26 xs2 iscsi-ha: 18651 This iteration is count 31 Sep 9 22:59:26 xs2 iscsi-ha: 18651 Checking if this host is a Pool Master or Slave Sep 9 22:59:26 xs2 kernel: [ 341.867446] connection1:0: detected conn error (1020) Sep 9 22:59:26 xs2 iscsi-ha: 18651 This host's pool status = slave:192.168.199.200 Sep 9 22:59:26 xs2 iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3) Sep 9 22:59:26 xs2 iscsi-ha: 18685 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 22:59:26 xs2 iscsi-ha: 18685 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 22:59:26 xs2 iscsi-ha: 18685 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- ns:0 nr:12292 dw:12292 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 22:59:26 xs2 iscsi-ha: 18685 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 22:59:27 xs2 iscsi-ha: 18685 DRBD Resource: iscsi1 in Connected state Sep 9 22:59:29 xs2 iscsid: connect to 10.10.10.3:3260 failed (Connection refused) Sep 9 22:59:32 xs2 ha-lizard: 3936 ha-lizard Watchdog: ha-lizard running - OK Sep 9 22:59:33 xs2 iscsi-ha-18646-INFO-/etc/iscsi-ha/iscsi-ha.sh: Scanning for Volume Group -> iscsi-sr: 4a13ab89-f5b5-e2e4-978b-73127607eae4 Sep 9 22:59:33 xs2 iscsi-ha-18646-INFO-/etc/iscsi-ha/iscsi-ha.sh: Volume Group for iSCSI-SR found OK: 4a13ab89-f5b5-e2e4-978b-73127607eae4 Sep 9 22:59:33 xs2 iscsid: connection1:0 is operational after recovery (2 attempts) Sep 9 22:59:33 xs2 iscsi-ha: 4150 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 22:59:36 xs2 iscsi-ha: 18643 Spawning new instance of iscsi-ha Sep 9 22:59:36 xs2 iscsi-ha: 18643 check_logger_processes Checking logger processes Sep 9 22:59:36 xs2 iscsi-ha: 18643 check_logger_processes No processes to clear Sep 9 22:59:36 xs2 iscsi-ha: 18825 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 22:59:36 xs2 iscsi-ha: 18825 This iteration is count 32 Sep 9 22:59:36 xs2 iscsi-ha: 18825 Checking if this host is a Pool Master or Slave Sep 9 22:59:36 xs2 iscsi-ha: 18825 This host's pool status = slave:192.168.199.200 Sep 9 22:59:36 xs2 iscsi-ha: 18859 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 22:59:37 xs2 iscsi-ha: 18859 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 22:59:37 xs2 iscsi-ha: 18859 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----- ns:0 nr:12292 dw:12292 dr:0 al:0 bm:1 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 22:59:37 xs2 iscsi-ha: 18859 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 22:59:37 xs2 iscsi-ha: 18859 DRBD Resource: iscsi1 in Connected state Sep 9 22:59:38 xs2 ha-lizard: 18349 Spawning new instance of ha-lizard Sep 9 22:59:39 xs2 ha-lizard: 18925 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 22:59:39 xs2 ha-lizard: 18925 This iteration is count 22 Sep 9 22:59:39 xs2 ha-lizard: 18925 Checking if this host is a Pool Master or Slave Sep 9 22:59:39 xs2 ha-lizard: 18925 This host's pool status = slave:192.168.199.200 Sep 9 22:59:42 xs2 ha-lizard: 18925 Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 9 22:59:42 xs2 ha-lizard: 18925 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Traceback (most recent call last): Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: message = smtplib.SMTP(smtp_server, smtp_port, hostname) Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 244, in __init__ Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: (code, msg) = self.connect(host, port) Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 306, in connect Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: raise socket.error, msg Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: socket.error: (111, 'Connection refused') Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xs2 Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Sep 9 22:59:42 EDT 2015 Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.8.7 Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Content: Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: xs2 Sep 9 22:59:42 xs2 ha-lizard: 18925 master_ip: Pool Master IP Address = 192.168.199.200 Sep 9 22:59:42 xs2 ha-lizard: 18925 This slave - xs2: 79769bcb-ad49-4ce1-8b27-51234c5b418f selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1 Sep 9 22:59:42 xs2 ha-lizard: 18925 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Traceback (most recent call last): Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xs2 Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: message = smtplib.SMTP(smtp_server, smtp_port, hostname) Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 244, in __init__ Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Sep 9 22:59:42 EDT 2015 Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: (code, msg) = self.connect(host, port) Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.8.7 Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 306, in connect Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: raise socket.error, msg Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: socket.error: (111, 'Connection refused') Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: xs2 Sep 9 22:59:42 xs2 ha-lizard: 18925 Pool Master NOT OK - Checking if ha-lizard is enabled in latest state file Sep 9 22:59:42 xs2 ha-lizard: 18925 Checking if ha-lizard is enabled Sep 9 22:59:42 xs2 ha-lizard: 18925 Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled Sep 9 22:59:42 xs2 ha-lizard: 18925 ha-lizard is enabled - continuing Sep 9 22:59:42 xs2 ha-lizard: 18925 Pool Master Monitor = Failed Sep 9 22:59:42 xs2 ha-lizard: 18925 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Traceback (most recent call last): Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: message = smtplib.SMTP(smtp_server, smtp_port, hostname) Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xs2 Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Sep 9 22:59:42 EDT 2015 Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 244, in __init__ Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.8.7 Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Content: Server xs2: Failed to contact pool master - manual intervention may be required Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: (code, msg) = self.connect(host, port) Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: xs2 Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 306, in connect Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: raise socket.error, msg Sep 9 22:59:42 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: socket.error: (111, 'Connection refused') Sep 9 22:59:42 xs2 ha-lizard: 18925 Retry Count set to 2. Retrying 2 times in 10 second intervals.. Sep 9 22:59:42 xs2 ha-lizard: 18925 Attempt 0: Checking Pool Master Status Sep 9 22:59:42 xs2 ha-lizard: 3936 ha-lizard Watchdog: ha-lizard running - OK Sep 9 22:59:43 xs2 iscsi-ha: 4150 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 22:59:46 xs2 iscsi-ha: 18817 iscsi-ha already running: Attempt 1 on PIDS: 18825 18820 18819 Sep 9 22:59:46 xs2 kernel: [ 362.307817] connection1:0: detected conn error (1020) Sep 9 22:59:47 xs2 iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3) Sep 9 22:59:49 xs2 iscsid: connect to 10.10.10.3:3260 failed (Connection refused) Sep 9 22:59:51 xs2 iscsi-ha: 18817 iscsi-ha already running: Attempt 2 on PIDS: 18825 18820 18819 Sep 9 22:59:52 xs2 ha-lizard: 18925 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 22:59:52 xs2 ha-lizard: 3936 ha-lizard Watchdog: ha-lizard running - OK Sep 9 22:59:52 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Traceback (most recent call last): Sep 9 22:59:52 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xs2 Sep 9 22:59:52 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: message = smtplib.SMTP(smtp_server, smtp_port, hostname) Sep 9 22:59:52 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 244, in __init__ Sep 9 22:59:52 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Sep 9 22:59:52 EDT 2015 Sep 9 22:59:52 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: (code, msg) = self.connect(host, port) Sep 9 22:59:52 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 306, in connect Sep 9 22:59:52 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.8.7 Sep 9 22:59:52 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: raise socket.error, msg Sep 9 22:59:52 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: socket.error: (111, 'Connection refused') Sep 9 22:59:52 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: xs2 Sep 9 22:59:53 xs2 ha-lizard: 18925 Attempt 1: Checking Pool Master Status Sep 9 22:59:53 xs2 iscsid: connect to 10.10.10.3:3260 failed (Connection refused) Sep 9 22:59:53 xs2 iscsi-ha: 4150 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 22:59:54 xs2 ha-lizard: 18917 ha-lizard already running: Attempt 1 on PIDS: 18925 18920 18919 18917 Sep 9 22:59:56 xs2 iscsi-ha: 18817 iscsi-ha already running: Attempt 3 on PIDS: 18825 18820 18819 Sep 9 22:59:57 xs2 iscsid: connect to 10.10.10.3:3260 failed (Connection refused) Sep 9 23:00:01 xs2 iscsi-ha: 18817 iscsi-ha already running: Attempt 4 on PIDS: 18825 18820 18819 Sep 9 23:00:02 xs2 kernel: [ 378.200187] d-con iscsi1: PingAck did not arrive in time. Sep 9 23:00:02 xs2 kernel: [ 378.200364] d-con iscsi1: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) Sep 9 23:00:02 xs2 kernel: [ 378.203419] d-con iscsi1: asender terminated Sep 9 23:00:02 xs2 kernel: [ 378.203490] d-con iscsi1: Terminating drbd_a_iscsi1 Sep 9 23:00:02 xs2 kernel: [ 378.204309] d-con iscsi1: Connection closed Sep 9 23:00:02 xs2 kernel: [ 378.204355] d-con iscsi1: conn( NetworkFailure -> Unconnected ) Sep 9 23:00:02 xs2 kernel: [ 378.204366] d-con iscsi1: receiver terminated Sep 9 23:00:02 xs2 kernel: [ 378.204371] d-con iscsi1: Restarting receiver thread Sep 9 23:00:02 xs2 kernel: [ 378.204374] d-con iscsi1: receiver (re)started Sep 9 23:00:02 xs2 kernel: [ 378.204387] d-con iscsi1: conn( Unconnected -> WFConnection ) Sep 9 23:00:02 xs2 ha-lizard: 3936 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:00:03 xs2 iscsi-ha: 4150 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:00:04 xs2 ha-lizard: 18917 ha-lizard already running: Attempt 2 on PIDS: 18925 18920 18919 18917 Sep 9 23:00:06 xs2 iscsi-ha: 18817 iscsi-ha already running: Attempt 5 on PIDS: 18825 18820 18819 At this point server restarted. Sep 9 23:03:06 xs2 iscsi-ha: 3796 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 23:03:06 xs2 iscsi-ha: 3796 This iteration is count 1 Sep 9 23:03:06 xs2 iscsi-ha: 3796 Checking if this host is a Pool Master or Slave Sep 9 23:03:06 xs2 iscsi-ha: 3796 This host's pool status = slave:192.168.199.200 Sep 9 23:03:06 xs2 iscsi-ha: 3861 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 23:03:06 xs2 iscsi-ha: 3861 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 23:03:06 xs2 iscsi-ha: 3861 DRBD not running - attempting start Sep 9 23:03:07 xs2 kernel: [ 163.130683] events: mcg drbd: 3 Sep 9 23:03:07 xs2 kernel: [ 163.147300] drbd: initialized. Version: 8.4.3 (api:1/proto:86-101) Sep 9 23:03:07 xs2 kernel: [ 163.147306] drbd: srcversion: 19422058F8A2D4AC0C8EF09 Sep 9 23:03:07 xs2 kernel: [ 163.147310] drbd: registered as block device major 147 Sep 9 23:03:07 xs2 iscsi-ha-3788-INFO-/etc/iscsi-ha/iscsi-ha.sh: Starting DRBD resources: [ Sep 9 23:03:07 xs2 iscsi-ha-3788-INFO-/etc/iscsi-ha/iscsi-ha.sh: create res: iscsi1 Sep 9 23:03:07 xs2 iscsi-ha-3788-INFO-/etc/iscsi-ha/iscsi-ha.sh: prepare disk: iscsi1 Sep 9 23:03:07 xs2 multipathd: drbd1: add path (uevent) Sep 9 23:03:07 xs2 multipathd: drbd1: failed to get path uid Sep 9 23:03:07 xs2 multipathd: uevent trigger error Sep 9 23:03:07 xs2 kernel: [ 163.207801] d-con iscsi1: Starting worker thread (from drbdsetup [3948]) Sep 9 23:03:07 xs2 kernel: klogd 1.4.1, ---------- state change ---------- Sep 9 23:03:07 xs2 kernel: [ 163.211757] block drbd1: disk( Diskless -> Attaching ) Sep 9 23:03:07 xs2 kernel: [ 163.212712] d-con iscsi1: Method to ensure write ordering: flush Sep 9 23:03:07 xs2 kernel: [ 163.212718] block drbd1: max BIO size = 1048576 Sep 9 23:03:07 xs2 kernel: [ 163.212726] block drbd1: drbd_bm_resize called with capacity == 37060064 Sep 9 23:03:07 xs2 kernel: [ 163.213348] block drbd1: resync bitmap: bits=4632508 words=72383 pages=142 Sep 9 23:03:07 xs2 kernel: [ 163.213354] block drbd1: size = 18 GB (18530032 KB) Sep 9 23:03:07 xs2 kernel: [ 163.222776] block drbd1: bitmap READ of 142 pages took 2 jiffies Sep 9 23:03:07 xs2 kernel: [ 163.223577] block drbd1: recounting of set bits took additional 0 jiffies Sep 9 23:03:07 xs2 kernel: [ 163.223583] block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map. Sep 9 23:03:07 xs2 kernel: [ 163.223592] block drbd1: disk( Attaching -> UpToDate ) Sep 9 23:03:07 xs2 kernel: [ 163.223598] block drbd1: attached to UUIDs 2A4D4897BD9B4848:0000000000000000:77E25E4498A81EC0:77E15E4498A81EC1 Sep 9 23:03:07 xs2 iscsi-ha-3788-INFO-/etc/iscsi-ha/iscsi-ha.sh: adjust disk: iscsi1 Sep 9 23:03:07 xs2 xenstored: A22 write /local/domain/0/vncterm-pid 3954 Sep 9 23:03:07 xs2 xenstored: A23 write /local/domain/0/console/vnc-port 5901 Sep 9 23:03:07 xs2 xenstored: A10 w event /local/domain/0/console/vnc-port /local/domain/0/console/vnc-port Sep 9 23:03:07 xs2 xenstored: A23 write /local/domain/0/console/tc-port 9501 Sep 9 23:03:07 xs2 xenstored: A10 w event /local/domain/0/console/tc-port /local/domain/0/console/tc-port Sep 9 23:03:07 xs2 modprobe: FATAL: Error inserting padlock_sha (/lib/modules/3.10.0+2/kernel/drivers/crypto/padlock-sha.ko): No such device Sep 9 23:03:07 xs2 kernel: [ 163.370696] d-con iscsi1: conn( StandAlone -> Unconnected ) Sep 9 23:03:07 xs2 kernel: [ 163.370770] d-con iscsi1: Starting receiver thread (from drbd_w_iscsi1 [3949]) Sep 9 23:03:07 xs2 iscsi-ha-3788-INFO-/etc/iscsi-ha/iscsi-ha.sh: adjust net: iscsi1 Sep 9 23:03:07 xs2 iscsi-ha-3788-INFO-/etc/iscsi-ha/iscsi-ha.sh: ] Sep 9 23:03:07 xs2 kernel: [ 163.375057] d-con iscsi1: receiver (re)started Sep 9 23:03:07 xs2 kernel: [ 163.375156] d-con iscsi1: conn( Unconnected -> WFConnection ) Sep 9 23:03:07 xs2 xsconsole: Started as /usr/lib64/xsconsole/XSConsole.py -f root Sep 9 23:03:08 xs2 iscsi-ha-3788-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 192.168.199.200 not responding - replug_pbd exiting Sep 9 23:03:08 xs2 xapi: [error|xs2|0 thread_zero|bringing up management interface D:2975a3008810|master_connection] Caught Xmlrpc_client.Connection_reset Sep 9 23:03:08 xs2 xcp-rrdd: [ warn|xs2|0 monitor|main|xmlrpc_client] stunnel pid: 3679 caught Xmlrpc_client.Connection_reset Sep 9 23:03:09 xs2 ha-lizard: 3702 Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 9 23:03:09 xs2 ha-lizard: 3702 This host has self fenced - HA forcefully disabled. Exiting... Sep 9 23:03:09 xs2 ha-lizard: 3702 HA can be reactiveted for this host by running /etc/ha-lizard/scripts/recover_fenced_host Sep 9 23:03:12 xs2 xapi: [ info|xs2|0 thread_zero|bringing up management interface D:2975a3008810|master_connection] stunnel connected pid=4976 fd=20 Sep 9 23:03:15 xs2 xapi: [error|xs2|0 thread_zero|bringing up management interface D:2975a3008810|master_connection] Caught Xmlrpc_client.Connection_reset Sep 9 23:03:16 xs2 ha-lizard: 3716 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:03:16 xs2 iscsi-ha: 3784 iscsi-ha already running: Attempt 1 on PIDS: 3796 3788 3787 3784 Sep 9 23:03:16 xs2 iscsi-ha: 3825 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:03:20 xs2 ha-lizard: 3690 Spawning new instance of ha-lizard Sep 9 23:03:21 xs2 ha-lizard: 6846 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:03:21 xs2 ha-lizard: 6846 This iteration is count 2 Sep 9 23:03:21 xs2 ha-lizard: 6846 Checking if this host is a Pool Master or Slave Sep 9 23:03:21 xs2 ha-lizard: 6846 This host's pool status = slave:192.168.199.200 Sep 9 23:03:21 xs2 iscsi-ha: 3784 iscsi-ha already running: Attempt 2 on PIDS: 3796 3788 3787 3784 Sep 9 23:03:22 xs2 xsconsole: No saved state available - using default state Sep 9 23:03:22 xs2 xsconsole: Loaded initial xapi and system data in 0.100 seconds Sep 9 23:03:23 xs2 xsconsole: Displaying 'XenAPI connection timeout' dialogue Sep 9 23:03:23 xs2 xapi: [ info|xs2|0 thread_zero|bringing up management interface D:2975a3008810|master_connection] stunnel connected pid=6916 fd=21 Sep 9 23:03:24 xs2 ha-lizard: 6846 Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 9 23:03:24 xs2 ha-lizard: 6846 This host has self fenced - HA forcefully disabled. Exiting... Sep 9 23:03:24 xs2 ha-lizard: 6846 HA can be reactiveted for this host by running /etc/ha-lizard/scripts/recover_fenced_host Sep 9 23:03:26 xs2 ha-lizard: 3716 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:03:26 xs2 iscsi-ha: 3784 iscsi-ha already running: Attempt 3 on PIDS: 3796 3788 3787 3784 Sep 9 23:03:26 xs2 iscsi-ha: 3825 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:03:26 xs2 xapi: [error|xs2|0 thread_zero|bringing up management interface D:2975a3008810|master_connection] Caught Xmlrpc_client.Connection_reset Sep 9 23:03:31 xs2 iscsi-ha: 3784 iscsi-ha already running: Attempt 4 on PIDS: 3796 3788 3787 3784 Sep 9 23:03:36 xs2 ha-lizard: 6837 Spawning new instance of ha-lizard Sep 9 23:03:36 xs2 ha-lizard: 3716 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:03:36 xs2 ha-lizard: 7033 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:03:36 xs2 ha-lizard: 7033 This iteration is count 3 Sep 9 23:03:36 xs2 ha-lizard: 7033 Checking if this host is a Pool Master or Slave Sep 9 23:03:36 xs2 ha-lizard: 7033 This host's pool status = slave:192.168.199.200 Sep 9 23:03:36 xs2 iscsi-ha: 3825 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:03:36 xs2 iscsi-ha: 3784 iscsi-ha already running: Attempt 5 on PIDS: 3796 3788 3787 3784 Sep 9 23:03:39 xs2 ha-lizard: 7033 Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 9 23:03:39 xs2 ha-lizard: 7033 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Traceback (most recent call last): Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: message = smtplib.SMTP(smtp_server, smtp_port, hostname) Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 244, in __init__ Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xs2 Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Sep 9 23:03:39 EDT 2015 Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.8.7 Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Content: Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: xs2 Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: (code, msg) = self.connect(host, port) Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 306, in connect Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: raise socket.error, msg Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: socket.error: (111, 'Connection refused') Sep 9 23:03:39 xs2 ha-lizard: 7033 This host has self fenced - HA forcefully disabled. Exiting... Sep 9 23:03:39 xs2 ha-lizard: 7033 HA can be reactiveted for this host by running /etc/ha-lizard/scripts/recover_fenced_host Sep 9 23:03:39 xs2 ha-lizard: 7033 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xs2 Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Sep 9 23:03:39 EDT 2015 Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Traceback (most recent call last): Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.8.7 Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Content: xs2: This host has self fenced - Recover with /etc/ha-lizard/scripts/recover_fenced_host Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: xs2 Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: message = smtplib.SMTP(smtp_server, smtp_port, hostname) Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 244, in __init__ Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: (code, msg) = self.connect(host, port) Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 306, in connect Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: raise socket.error, msg Sep 9 23:03:39 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: socket.error: (111, 'Connection refused') Sep 9 23:03:41 xs2 iscsi-ha: 3784 iscsi-ha already running: Attempt 6 on PIDS: 3796 3788 3787 3784 Sep 9 23:03:41 xs2 iscsi-ha: 3784 iscsi-ha failed to spawn new instance after 6 attmepts. Check xs2 for possible hung process Sep 9 23:03:41 xs2 iscsi-ha: 3784 MONITOR has reached configured threshhold - 5 - MONITOR_KILLALL is enabled. - attempting to kill all existing processes Sep 9 23:03:41 xs2 iscsi-ha-NOTICE-/etc/rc3.d/S99iscsi-ha: /etc/rc3.d/S99iscsi-ha: line 34: 3784 Terminated /etc/iscsi-ha/iscsi-ha.sh Sep 9 23:03:41 xs2 iscsi-ha: 3784 Successfully killed all instances of iscsi-ha.sh Sep 9 23:03:42 xs2 xapi: [ info|xs2|0 thread_zero|bringing up management interface D:2975a3008810|master_connection] stunnel connected pid=7147 fd=22 Sep 9 23:03:45 xs2 xapi: [error|xs2|0 thread_zero|bringing up management interface D:2975a3008810|master_connection] Caught Xmlrpc_client.Connection_reset Sep 9 23:03:46 xs2 ha-lizard: 3716 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:03:46 xs2 iscsi-ha: 3825 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:03:46 xs2 iscsi-ha: 3784 Spawning new instance of iscsi-ha Sep 9 23:03:46 xs2 iscsi-ha: 3784 check_logger_processes Checking logger processes Sep 9 23:03:46 xs2 iscsi-ha: 3784 check_logger_processes No processes to clear Sep 9 23:03:46 xs2 iscsi-ha: 7197 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 23:03:46 xs2 iscsi-ha: 7197 This iteration is count 2 Sep 9 23:03:46 xs2 iscsi-ha: 7197 Checking if this host is a Pool Master or Slave Sep 9 23:03:46 xs2 iscsi-ha: 7197 This host's pool status = slave:192.168.199.200 Sep 9 23:03:47 xs2 iscsi-ha: 7231 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 23:03:47 xs2 iscsi-ha: 7231 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 23:03:47 xs2 iscsi-ha: 7231 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 23:03:47 xs2 iscsi-ha: 7231 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 23:03:47 xs2 iscsi-ha: 7231 DRBD Resource: iscsi1 in WFConnection state - expected Connected state Sep 9 23:03:48 xs2 iscsi-ha-7192-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 192.168.199.200 not responding - replug_pbd exiting Sep 9 23:03:51 xs2 ha-lizard: 7025 Spawning new instance of ha-lizard Sep 9 23:03:51 xs2 ha-lizard: 7314 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:03:51 xs2 ha-lizard: 7314 This iteration is count 4 Sep 9 23:03:51 xs2 ha-lizard: 7314 Checking if this host is a Pool Master or Slave Sep 9 23:03:51 xs2 ha-lizard: 7314 This host's pool status = slave:192.168.199.200 Sep 9 23:03:54 xs2 ha-lizard: 7314 Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 9 23:03:54 xs2 ha-lizard: 7314 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:03:54 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Traceback (most recent call last): Sep 9 23:03:54 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: message = smtplib.SMTP(smtp_server, smtp_port, hostname) Sep 9 23:03:54 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 244, in __init__ Sep 9 23:03:54 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xs2 Sep 9 23:03:54 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: (code, msg) = self.connect(host, port) Sep 9 23:03:54 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: File "/usr/lib64/python2.4/smtplib.py", line 306, in connect Sep 9 23:03:54 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Timestamp: Wed Sep 9 23:03:54 EDT 2015 Sep 9 23:03:54 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: raise socket.error, msg Sep 9 23:03:54 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Process: ha-lizard Version: 1.8.7 Sep 9 23:03:54 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: socket.error: (111, 'Connection refused') Sep 9 23:03:54 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Content: Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 9 23:03:54 xs2 ha-lizard-NOTICE-/etc/ha-lizard/ha-lizard.sh: Email Alert Message Hostname: xs2 Sep 9 23:03:54 xs2 ha-lizard: 7314 This host has self fenced - HA forcefully disabled. Exiting... Sep 9 23:03:54 xs2 ha-lizard: 7314 HA can be reactiveted for this host by running /etc/ha-lizard/scripts/recover_fenced_host Sep 9 23:03:54 xs2 ha-lizard: 7314 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:03:56 xs2 ha-lizard: 3716 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:03:56 xs2 iscsi-ha: 3825 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:03:56 xs2 iscsi-ha: 7189 Spawning new instance of iscsi-ha Sep 9 23:03:56 xs2 iscsi-ha: 7189 check_logger_processes Checking logger processes Sep 9 23:03:56 xs2 iscsi-ha: 7189 check_logger_processes No processes to clear Sep 9 23:03:57 xs2 iscsi-ha: 7433 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 23:03:57 xs2 iscsi-ha: 7433 This iteration is count 3 Sep 9 23:03:57 xs2 iscsi-ha: 7433 Checking if this host is a Pool Master or Slave Sep 9 23:03:57 xs2 iscsi-ha: 7433 This host's pool status = slave:192.168.199.200 Sep 9 23:03:57 xs2 iscsi-ha: 7467 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 23:03:57 xs2 iscsi-ha: 7467 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 23:03:57 xs2 iscsi-ha: 7467 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 23:03:57 xs2 iscsi-ha: 7467 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 23:03:57 xs2 iscsi-ha: 7467 DRBD Resource: iscsi1 in WFConnection state - expected Connected state Sep 9 23:04:00 xs2 iscsi-ha-7428-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 192.168.199.200 not responding - replug_pbd exiting Sep 9 23:04:06 xs2 ha-lizard: 3716 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:04:06 xs2 ha-lizard: 7306 Spawning new instance of ha-lizard Sep 9 23:04:06 xs2 ha-lizard: 7613 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:04:06 xs2 ha-lizard: 7613 This iteration is count 5 Sep 9 23:04:06 xs2 ha-lizard: 7613 Checking if this host is a Pool Master or Slave Sep 9 23:04:06 xs2 ha-lizard: 7613 This host's pool status = slave:192.168.199.200 Sep 9 23:04:06 xs2 iscsi-ha: 3825 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:04:07 xs2 iscsi-ha: 7425 Spawning new instance of iscsi-ha Sep 9 23:04:07 xs2 iscsi-ha: 7425 check_logger_processes Checking logger processes Sep 9 23:04:07 xs2 iscsi-ha: 7425 check_logger_processes No processes to clear Sep 9 23:04:07 xs2 iscsi-ha: 7654 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 23:04:07 xs2 iscsi-ha: 7654 This iteration is count 4 Sep 9 23:04:07 xs2 iscsi-ha: 7654 Checking if this host is a Pool Master or Slave Sep 9 23:04:07 xs2 iscsi-ha: 7654 This host's pool status = slave:192.168.199.200 Sep 9 23:04:07 xs2 iscsi-ha: 7694 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 23:04:07 xs2 iscsi-ha: 7694 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 23:04:07 xs2 iscsi-ha: 7694 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 23:04:07 xs2 iscsi-ha: 7694 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 23:04:07 xs2 iscsi-ha: 7694 DRBD Resource: iscsi1 in WFConnection state - expected Connected state Sep 9 23:04:09 xs2 ha-lizard: 7613 Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 9 23:04:09 xs2 ha-lizard: 7613 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:04:09 xs2 ha-lizard: 7613 This host has self fenced - HA forcefully disabled. Exiting... Sep 9 23:04:09 xs2 ha-lizard: 7613 HA can be reactiveted for this host by running /etc/ha-lizard/scripts/recover_fenced_host Sep 9 23:04:09 xs2 ha-lizard: 7613 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:04:10 xs2 iscsi-ha-7649-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 192.168.199.200 not responding - replug_pbd exiting Sep 9 23:04:16 xs2 ha-lizard: 3716 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:04:16 xs2 iscsi-ha: 3825 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:04:17 xs2 iscsi-ha: 7646 Spawning new instance of iscsi-ha Sep 9 23:04:17 xs2 iscsi-ha: 7646 check_logger_processes Checking logger processes Sep 9 23:04:17 xs2 iscsi-ha: 7646 check_logger_processes No processes to clear Sep 9 23:04:17 xs2 iscsi-ha: 7857 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 23:04:17 xs2 iscsi-ha: 7857 This iteration is count 5 Sep 9 23:04:17 xs2 iscsi-ha: 7857 Checking if this host is a Pool Master or Slave Sep 9 23:04:17 xs2 iscsi-ha: 7857 This host's pool status = slave:192.168.199.200 Sep 9 23:04:17 xs2 iscsi-ha: 7898 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 23:04:17 xs2 iscsi-ha: 7898 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 23:04:17 xs2 iscsi-ha: 7898 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 23:04:17 xs2 iscsi-ha: 7898 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 23:04:17 xs2 iscsi-ha: 7898 DRBD Resource: iscsi1 in WFConnection state - expected Connected state Sep 9 23:04:17 xs2 iscsi-ha-NOTICE-/etc/iscsi-ha/iscsi-ha.sh: cat: /dev/shm/iscsi-ha-mail/*.msg: No such file or directory Sep 9 23:04:17 xs2 iscsi-ha-NOTICE-/etc/iscsi-ha/iscsi-ha.sh: /etc/iscsi-ha/iscsi-ha.func: line 283: [: -eq: unary operator expected Sep 9 23:04:17 xs2 iscsi-ha-NOTICE-/etc/iscsi-ha/iscsi-ha.sh: Traceback (most recent call last): Sep 9 23:04:17 xs2 iscsi-ha-NOTICE-/etc/iscsi-ha/iscsi-ha.sh: message = smtplib.SMTP(smtp_server, smtp_port, hostname) Sep 9 23:04:17 xs2 iscsi-ha-7852-INFO-/etc/iscsi-ha/iscsi-ha.sh: Email Alert Subject: HA-Lizard noSAN SYSTEM ALERT - FROM HOST: xs2 Sep 9 23:04:17 xs2 iscsi-ha-7852-INFO-/etc/iscsi-ha/iscsi-ha.sh: Email Alert Timestamp: Wed Sep 9 23:04:17 EDT 2015 Sep 9 23:04:17 xs2 iscsi-ha-7852-INFO-/etc/iscsi-ha/iscsi-ha.sh: Email Alert Process: iscsi-ha Version: IHA_1.5.5 Sep 9 23:04:17 xs2 iscsi-ha-NOTICE-/etc/iscsi-ha/iscsi-ha.sh: File "/usr/lib64/python2.4/smtplib.py", line 244, in __init__ Sep 9 23:04:17 xs2 iscsi-ha-7852-INFO-/etc/iscsi-ha/iscsi-ha.sh: Email Alert Message Content: DRBD Resource: iscsi1 in WFConnection state - expected Connected state Sep 9 23:04:17 xs2 iscsi-ha-7852-INFO-/etc/iscsi-ha/iscsi-ha.sh: Email Alert Message Hostname: xs2 Sep 9 23:04:17 xs2 iscsi-ha-NOTICE-/etc/iscsi-ha/iscsi-ha.sh: (code, msg) = self.connect(host, port) Sep 9 23:04:17 xs2 iscsi-ha-NOTICE-/etc/iscsi-ha/iscsi-ha.sh: File "/usr/lib64/python2.4/smtplib.py", line 306, in connect Sep 9 23:04:17 xs2 iscsi-ha-NOTICE-/etc/iscsi-ha/iscsi-ha.sh: raise socket.error, msg Sep 9 23:04:17 xs2 iscsi-ha-NOTICE-/etc/iscsi-ha/iscsi-ha.sh: socket.error: (111, 'Connection refused') Sep 9 23:04:17 xs2 xapi: [ info|xs2|0 thread_zero|bringing up management interface D:2975a3008810|master_connection] stunnel connected pid=7952 fd=24 Sep 9 23:04:20 xs2 iscsi-ha-7852-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 192.168.199.200 not responding - replug_pbd exiting Sep 9 23:04:20 xs2 xapi: [error|xs2|0 thread_zero|bringing up management interface D:2975a3008810|master_connection] Caught Xmlrpc_client.Connection_reset Sep 9 23:04:21 xs2 ha-lizard: 7605 Spawning new instance of ha-lizard Sep 9 23:04:21 xs2 ha-lizard: 7995 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:04:21 xs2 ha-lizard: 7995 This iteration is count 6 Sep 9 23:04:21 xs2 ha-lizard: 7995 Checking if this host is a Pool Master or Slave Sep 9 23:04:21 xs2 ha-lizard: 7995 This host's pool status = slave:192.168.199.200 Sep 9 23:04:24 xs2 ha-lizard: 7995 Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 9 23:04:24 xs2 ha-lizard: 7995 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:04:24 xs2 ha-lizard: 7995 This host has self fenced - HA forcefully disabled. Exiting... Sep 9 23:04:24 xs2 ha-lizard: 7995 HA can be reactiveted for this host by running /etc/ha-lizard/scripts/recover_fenced_host Sep 9 23:04:24 xs2 ha-lizard: 7995 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:04:26 xs2 ha-lizard: 3716 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:04:26 xs2 iscsi-ha: 3825 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:04:27 xs2 iscsi-ha: 7849 Spawning new instance of iscsi-ha Sep 9 23:04:27 xs2 iscsi-ha: 7849 check_logger_processes Checking logger processes Sep 9 23:04:27 xs2 iscsi-ha: 7849 check_logger_processes No processes to clear Sep 9 23:04:27 xs2 iscsi-ha: 8107 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 23:04:27 xs2 iscsi-ha: 8107 This iteration is count 6 Sep 9 23:04:27 xs2 iscsi-ha: 8107 Checking if this host is a Pool Master or Slave Sep 9 23:04:27 xs2 iscsi-ha: 8107 This host's pool status = slave:192.168.199.200 Sep 9 23:04:27 xs2 iscsi-ha: 8148 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 23:04:27 xs2 iscsi-ha: 8148 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 23:04:27 xs2 iscsi-ha: 8148 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 23:04:27 xs2 iscsi-ha: 8148 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 23:04:27 xs2 iscsi-ha: 8148 DRBD Resource: iscsi1 in WFConnection state - expected Connected state Sep 9 23:04:30 xs2 iscsi-ha-8102-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 192.168.199.200 not responding - replug_pbd exiting Sep 9 23:04:36 xs2 ha-lizard: 3716 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:04:36 xs2 ha-lizard: 7987 Spawning new instance of ha-lizard Sep 9 23:04:36 xs2 ha-lizard: 8270 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:04:36 xs2 ha-lizard: 8270 This iteration is count 7 Sep 9 23:04:36 xs2 ha-lizard: 8270 Checking if this host is a Pool Master or Slave Sep 9 23:04:36 xs2 ha-lizard: 8270 This host's pool status = slave:192.168.199.200 Sep 9 23:04:36 xs2 iscsi-ha: 3825 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:04:37 xs2 iscsi-ha: 8099 Spawning new instance of iscsi-ha Sep 9 23:04:37 xs2 iscsi-ha: 8099 check_logger_processes Checking logger processes Sep 9 23:04:37 xs2 iscsi-ha: 8099 check_logger_processes No processes to clear Sep 9 23:04:37 xs2 iscsi-ha: 8320 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 23:04:37 xs2 iscsi-ha: 8320 This iteration is count 7 Sep 9 23:04:37 xs2 iscsi-ha: 8320 Checking if this host is a Pool Master or Slave Sep 9 23:04:37 xs2 iscsi-ha: 8320 This host's pool status = slave:192.168.199.200 Sep 9 23:04:37 xs2 iscsi-ha: 8361 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 23:04:37 xs2 iscsi-ha: 8361 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 23:04:37 xs2 iscsi-ha: 8361 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 23:04:37 xs2 iscsi-ha: 8361 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 23:04:37 xs2 iscsi-ha: 8361 DRBD Resource: iscsi1 in WFConnection state - expected Connected state Sep 9 23:04:39 xs2 ha-lizard: 8270 Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 9 23:04:39 xs2 ha-lizard: 8270 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:04:39 xs2 ha-lizard: 8270 This host has self fenced - HA forcefully disabled. Exiting... Sep 9 23:04:39 xs2 ha-lizard: 8270 HA can be reactiveted for this host by running /etc/ha-lizard/scripts/recover_fenced_host Sep 9 23:04:39 xs2 ha-lizard: 8270 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:04:40 xs2 iscsi-ha-8315-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 192.168.199.200 not responding - replug_pbd exiting Sep 9 23:04:46 xs2 ha-lizard: 3716 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:04:46 xs2 iscsi-ha: 3825 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:04:47 xs2 iscsi-ha: 8312 Spawning new instance of iscsi-ha Sep 9 23:04:47 xs2 iscsi-ha: 8312 check_logger_processes Checking logger processes Sep 9 23:04:47 xs2 iscsi-ha: 8312 check_logger_processes No processes to clear Sep 9 23:04:47 xs2 iscsi-ha: 8531 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 23:04:47 xs2 iscsi-ha: 8531 This iteration is count 8 Sep 9 23:04:47 xs2 iscsi-ha: 8531 Checking if this host is a Pool Master or Slave Sep 9 23:04:47 xs2 iscsi-ha: 8531 This host's pool status = slave:192.168.199.200 Sep 9 23:04:47 xs2 iscsi-ha: 8566 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 23:04:47 xs2 iscsi-ha: 8566 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 23:04:47 xs2 iscsi-ha: 8566 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 23:04:48 xs2 iscsi-ha: 8566 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 23:04:48 xs2 iscsi-ha: 8566 DRBD Resource: iscsi1 in WFConnection state - expected Connected state Sep 9 23:04:50 xs2 iscsi-ha-8526-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 192.168.199.200 not responding - replug_pbd exiting Sep 9 23:04:51 xs2 ha-lizard: 8262 Spawning new instance of ha-lizard Sep 9 23:04:51 xs2 ha-lizard: 8685 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:04:51 xs2 ha-lizard: 8685 This iteration is count 8 Sep 9 23:04:51 xs2 ha-lizard: 8685 Checking if this host is a Pool Master or Slave Sep 9 23:04:51 xs2 ha-lizard: 8685 This host's pool status = slave:192.168.199.200 Sep 9 23:04:54 xs2 ha-lizard: 8685 Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 9 23:04:54 xs2 ha-lizard: 8685 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:04:54 xs2 ha-lizard: 8685 This host has self fenced - HA forcefully disabled. Exiting... Sep 9 23:04:54 xs2 ha-lizard: 8685 HA can be reactiveted for this host by running /etc/ha-lizard/scripts/recover_fenced_host Sep 9 23:04:54 xs2 ha-lizard: 8685 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:04:56 xs2 ha-lizard: 3716 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:04:56 xs2 iscsi-ha: 3825 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:04:57 xs2 iscsi-ha: 8523 Spawning new instance of iscsi-ha Sep 9 23:04:57 xs2 iscsi-ha: 8523 check_logger_processes Checking logger processes Sep 9 23:04:57 xs2 iscsi-ha: 8523 check_logger_processes No processes to clear Sep 9 23:04:57 xs2 iscsi-ha: 8812 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 23:04:57 xs2 iscsi-ha: 8812 This iteration is count 9 Sep 9 23:04:57 xs2 iscsi-ha: 8812 Checking if this host is a Pool Master or Slave Sep 9 23:04:57 xs2 iscsi-ha: 8812 This host's pool status = slave:192.168.199.200 Sep 9 23:04:57 xs2 iscsi-ha: 8847 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 23:04:58 xs2 iscsi-ha: 8847 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 23:04:58 xs2 iscsi-ha: 8847 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 23:04:58 xs2 iscsi-ha: 8847 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 23:04:58 xs2 iscsi-ha: 8847 DRBD Resource: iscsi1 in WFConnection state - expected Connected state Sep 9 23:05:01 xs2 iscsi-ha-8807-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 192.168.199.200 not responding - replug_pbd exiting Sep 9 23:05:06 xs2 ha-lizard: 3716 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:05:06 xs2 ha-lizard: 8677 Spawning new instance of ha-lizard Sep 9 23:05:06 xs2 iscsi-ha: 3825 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:05:06 xs2 ha-lizard: 8993 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:05:06 xs2 ha-lizard: 8993 This iteration is count 9 Sep 9 23:05:06 xs2 ha-lizard: 8993 Checking if this host is a Pool Master or Slave Sep 9 23:05:06 xs2 ha-lizard: 8993 This host's pool status = slave:192.168.199.200 Sep 9 23:05:07 xs2 iscsi-ha: 8804 Spawning new instance of iscsi-ha Sep 9 23:05:07 xs2 iscsi-ha: 8804 check_logger_processes Checking logger processes Sep 9 23:05:07 xs2 iscsi-ha: 8804 check_logger_processes No processes to clear Sep 9 23:05:07 xs2 iscsi-ha: 9040 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 23:05:07 xs2 iscsi-ha: 9040 This iteration is count 10 Sep 9 23:05:07 xs2 iscsi-ha: 9040 Checking if this host is a Pool Master or Slave Sep 9 23:05:07 xs2 iscsi-ha: 9040 This host's pool status = slave:192.168.199.200 Sep 9 23:05:08 xs2 iscsi-ha: 9075 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 23:05:08 xs2 iscsi-ha: 9075 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 23:05:08 xs2 iscsi-ha: 9075 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 23:05:08 xs2 iscsi-ha: 9075 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 23:05:08 xs2 iscsi-ha: 9075 DRBD Resource: iscsi1 in WFConnection state - expected Connected state Sep 9 23:05:09 xs2 ha-lizard: 8993 Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 9 23:05:09 xs2 ha-lizard: 8993 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:05:09 xs2 ha-lizard: 8993 This host has self fenced - HA forcefully disabled. Exiting... Sep 9 23:05:10 xs2 ha-lizard: 8993 HA can be reactiveted for this host by running /etc/ha-lizard/scripts/recover_fenced_host Sep 9 23:05:10 xs2 ha-lizard: 8993 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 9 23:05:11 xs2 iscsi-ha-9035-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 192.168.199.200 not responding - replug_pbd exiting Sep 9 23:05:16 xs2 ha-lizard: 3716 ha-lizard Watchdog: ha-lizard running - OK Sep 9 23:05:16 xs2 iscsi-ha: 3825 iscsi-ha Watchdog: iscsi-ha running - OK Sep 9 23:05:17 xs2 iscsi-ha: 9032 Spawning new instance of iscsi-ha Sep 9 23:05:17 xs2 iscsi-ha: 9032 check_logger_processes Checking logger processes Sep 9 23:05:17 xs2 iscsi-ha: 9032 check_logger_processes No processes to clear Sep 9 23:05:17 xs2 iscsi-ha: 9246 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 9 23:05:17 xs2 iscsi-ha: 9246 This iteration is count 11 Sep 9 23:05:17 xs2 iscsi-ha: 9246 Checking if this host is a Pool Master or Slave Sep 9 23:05:17 xs2 iscsi-ha: 9246 This host's pool status = slave:192.168.199.200 Sep 9 23:05:18 xs2 iscsi-ha: 9286 local_ip_list: Local IP list returned 127.0.0.1 192.168.199.159 10.10.10.2 Sep 9 23:05:18 xs2 iscsi-ha: 9286 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Sep 9 23:05:18 xs2 iscsi-ha: 9286 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: 19422058F8A2D4AC0C8EF09 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 9 23:05:18 xs2 iscsi-ha: 9286 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 9 23:05:18 xs2 iscsi-ha: 9286 DRBD Resource: iscsi1 in WFConnection state - expected Connected state