Jul 4 14:46:20 169.0.0.71 ovs-vswitchd: 00035|bond|INFO|interface eth3: link state down Jul 4 14:46:20 169.0.0.71 ovs-vswitchd: 00036|bond|INFO|interface eth3: will be disabled if it stays down for 200 ms Jul 4 14:46:20 169.0.0.71 ovs-vswitchd: 00036|bond|INFO|interface eth3: will be disabled if it stays down for 200 ms Jul 4 14:46:20 169.0.0.71 ovs-vswitchd: 00037|bond|WARN|interface eth3: disabled Jul 4 14:46:20 169.0.0.71 ovs-vswitchd: 00037|bond|WARN|interface eth3: disabled Jul 4 14:46:21 169.0.0.71 kernel: [ 229.320619] igb: eth2 NIC Link is Down Jul 4 14:46:21 169.0.0.71 kernel: [ 229.320619] igb: eth2 NIC Link is Down Jul 4 14:46:21 169.0.0.71 xcp-networkd: [ info|XCP2|1||network_monitor_thread] Bonds status changed: bond0 nb_links 2 up 1 up_old 2 Jul 4 14:46:21 169.0.0.71 xcp-networkd: [ info|XCP2|1||network_monitor_thread] Bonds status changed: bond0 nb_links 2 up 1 up_old 2 Jul 4 14:46:21 169.0.0.71 ovs-vswitchd: 00038|bond|INFO|interface eth2: link state down Jul 4 14:46:21 169.0.0.71 ovs-vswitchd: 00038|bond|INFO|interface eth2: link state down Jul 4 14:46:21 169.0.0.71 ovs-vswitchd: 00039|bond|INFO|interface eth2: will be disabled if it stays down for 200 ms Jul 4 14:46:21 169.0.0.71 ovs-vswitchd: 00039|bond|INFO|interface eth2: will be disabled if it stays down for 200 ms Jul 4 14:46:21 169.0.0.71 ovs-vswitchd: 00040|bond|WARN|interface eth2: disabled Jul 4 14:46:21 169.0.0.71 ovs-vswitchd: 00040|bond|WARN|interface eth2: disabled Jul 4 14:46:21 169.0.0.71 ovs-vswitchd: 00041|bond|WARN|bond bond0: all interfaces disabled Jul 4 14:46:21 169.0.0.71 ovs-vswitchd: 00041|bond|WARN|bond bond0: all interfaces disabled Jul 4 14:46:22 169.0.0.71 iscsi-ha: 16647 iscsi-ha already running: Attempt 4 on PIDS: 16655 16650 16649 Jul 4 14:46:22 169.0.0.71 iscsi-ha: 16647 iscsi-ha already running: Attempt 4 on PIDS: 16655 16650 16649 Jul 4 14:46:23 169.0.0.71 xapi: [ warn|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|xmlrpc_client] stunnel pid: 16966 caught Xmlrpc_client.Connection_reset Jul 4 14:46:23 169.0.0.71 xapi: [ warn|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|xmlrpc_client] stunnel pid: 16966 caught Xmlrpc_client.Connection_reset Jul 4 14:46:23 169.0.0.71 xapi: [error|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] Caught Unix.Unix_error(31, "write", "") Jul 4 14:46:23 169.0.0.71 xapi: [error|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] Caught Unix.Unix_error(31, "write", "") Jul 4 14:46:23 169.0.0.71 xapi: [error|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message). Jul 4 14:46:23 169.0.0.71 xapi: [error|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message). Jul 4 14:46:25 169.0.0.71 xapi: [ info|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] stunnel connected pid=17073 fd=22 Jul 4 14:46:25 169.0.0.71 xapi: [ info|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] stunnel connected pid=17073 fd=22 Jul 4 14:46:26 169.0.0.71 ha-lizard: 6957 ha-lizard Watchdog: ha-lizard running - OK Jul 4 14:46:26 169.0.0.71 ha-lizard: 6957 ha-lizard Watchdog: ha-lizard running - OK Jul 4 14:46:26 169.0.0.71 iscsi-ha: 7109 iscsi-ha Watchdog: iscsi-ha running - OK Jul 4 14:46:26 169.0.0.71 iscsi-ha: 7109 iscsi-ha Watchdog: iscsi-ha running - OK Jul 4 14:46:26 169.0.0.71 ha-lizard: 16816 ha-lizard already running: Attempt 2 on PIDS: 16824 16819 16818 16816 Jul 4 14:46:26 169.0.0.71 ha-lizard: 16816 ha-lizard already running: Attempt 2 on PIDS: 16824 16819 16818 16816 Jul 4 14:46:27 169.0.0.71 iscsi-ha: 16647 iscsi-ha already running: Attempt 5 on PIDS: 16655 16650 16649 Jul 4 14:46:27 169.0.0.71 iscsi-ha: 16647 iscsi-ha already running: Attempt 5 on PIDS: 16655 16650 16649 Jul 4 14:46:28 169.0.0.71 ntpd[6281]: synchronized to 149.156.70.5, stratum 2 Jul 4 14:46:28 169.0.0.71 ntpd[6281]: synchronized to 149.156.70.5, stratum 2 Jul 4 14:46:32 169.0.0.71 iscsi-ha: 16647 iscsi-ha already running: Attempt 6 on PIDS: 16655 16650 16649 Jul 4 14:46:32 169.0.0.71 iscsi-ha: 16647 iscsi-ha already running: Attempt 6 on PIDS: 16655 16650 16649 Jul 4 14:46:32 169.0.0.71 iscsi-ha: 16647 iscsi-ha failed to spawn new instance after 6 attmepts. Check XCP2 for possible hung process Jul 4 14:46:32 169.0.0.71 iscsi-ha: 16647 iscsi-ha failed to spawn new instance after 6 attmepts. Check XCP2 for possible hung process Jul 4 14:46:32 169.0.0.71 iscsi-ha: 16647 MONITOR has reached configured threshhold - 5 - MONITOR_KILLALL is enabled. - attempting to kill all existing processes Jul 4 14:46:32 169.0.0.71 iscsi-ha: 16647 MONITOR has reached configured threshhold - 5 - MONITOR_KILLALL is enabled. - attempting to kill all existing processes Jul 4 14:46:32 169.0.0.71 iscsi-ha: 16647 Successfully killed all instances of iscsi-ha.sh Jul 4 14:46:32 169.0.0.71 iscsi-ha: 16647 Successfully killed all instances of iscsi-ha.sh Jul 4 14:46:34 169.0.0.71 ha-lizard: 16824 check_xapi: Pool Host 169.0.0.70 failed to respond to ICMP ping Jul 4 14:46:34 169.0.0.71 ha-lizard: 16824 check_xapi: Pool Host 169.0.0.70 failed to respond to ICMP ping Jul 4 14:46:34 169.0.0.71 ha-lizard: 16824 Attempt 1: Checking Pool Master Status Jul 4 14:46:34 169.0.0.71 ha-lizard: 16824 Attempt 1: Checking Pool Master Status Jul 4 14:46:35 169.0.0.71 xapi: [ warn|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|xmlrpc_client] stunnel pid: 17076 caught Xmlrpc_client.Connection_reset Jul 4 14:46:35 169.0.0.71 xapi: [ warn|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|xmlrpc_client] stunnel pid: 17076 caught Xmlrpc_client.Connection_reset Jul 4 14:46:36 169.0.0.71 xapi: [error|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] Caught Unix.Unix_error(31, "write", "") Jul 4 14:46:36 169.0.0.71 xapi: [error|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] Caught Unix.Unix_error(31, "write", "") Jul 4 14:46:36 169.0.0.71 xapi: [error|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message). Jul 4 14:46:36 169.0.0.71 xapi: [error|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message). Jul 4 14:46:36 169.0.0.71 ha-lizard: 6957 ha-lizard Watchdog: ha-lizard running - OK Jul 4 14:46:36 169.0.0.71 ha-lizard: 6957 ha-lizard Watchdog: ha-lizard running - OK Jul 4 14:46:36 169.0.0.71 iscsi-ha: 7109 iscsi-ha Watchdog: iscsi-ha running - OK Jul 4 14:46:36 169.0.0.71 iscsi-ha: 7109 iscsi-ha Watchdog: iscsi-ha running - OK Jul 4 14:46:36 169.0.0.71 ha-lizard: 16816 ha-lizard already running: Attempt 3 on PIDS: 16824 16819 16818 16816 Jul 4 14:46:36 169.0.0.71 ha-lizard: 16816 ha-lizard already running: Attempt 3 on PIDS: 16824 16819 16818 16816 Jul 4 14:46:37 169.0.0.71 xapi: [error|XCP2|218|xapi events D:4f1f6d246b2a|stunnel] get_reusable_stunnel: fresh stunnel failed reusable check; delaying 10.00 seconds before reconnecting to 169.0.0.70:443 (attempt 3 / 10) Jul 4 14:46:37 169.0.0.71 xapi: [error|XCP2|218|xapi events D:4f1f6d246b2a|stunnel] get_reusable_stunnel: fresh stunnel failed reusable check; delaying 10.00 seconds before reconnecting to 169.0.0.70:443 (attempt 3 / 10) Jul 4 14:46:37 169.0.0.71 iscsi-ha: 16647 Spawning new instance of iscsi-ha Jul 4 14:46:37 169.0.0.71 iscsi-ha: 16647 Spawning new instance of iscsi-ha Jul 4 14:46:37 169.0.0.71 iscsi-ha: 16647 check_logger_processes Checking logger processes Jul 4 14:46:37 169.0.0.71 iscsi-ha: 16647 check_logger_processes Checking logger processes Jul 4 14:46:37 169.0.0.71 iscsi-ha: 16647 check_logger_processes No processes to clear Jul 4 14:46:37 169.0.0.71 iscsi-ha: 16647 check_logger_processes No processes to clear Jul 4 14:46:37 169.0.0.71 iscsi-ha: 17203 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Jul 4 14:46:37 169.0.0.71 iscsi-ha: 17203 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Jul 4 14:46:37 169.0.0.71 iscsi-ha: 17203 This iteration is count 18 Jul 4 14:46:37 169.0.0.71 iscsi-ha: 17203 This iteration is count 18 Jul 4 14:46:37 169.0.0.71 iscsi-ha: 17203 Checking if this host is a Pool Master or Slave Jul 4 14:46:37 169.0.0.71 iscsi-ha: 17203 Checking if this host is a Pool Master or Slave Jul 4 14:46:37 169.0.0.71 iscsi-ha: 17203 This host's pool status = slave:169.0.0.70 Jul 4 14:46:37 169.0.0.71 iscsi-ha: 17203 This host's pool status = slave:169.0.0.70 Jul 4 14:46:37 169.0.0.71 xapi: [ info|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] stunnel connected pid=17216 fd=22 Jul 4 14:46:37 169.0.0.71 xapi: [ info|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] stunnel connected pid=17216 fd=22 Jul 4 14:46:38 169.0.0.71 iscsi-ha: 17245 local_ip_list: Local IP list returned 127.0.0.1 169.0.0.71 10.10.10.2 Jul 4 14:46:38 169.0.0.71 iscsi-ha: 17245 local_ip_list: Local IP list returned 127.0.0.1 169.0.0.71 10.10.10.2 Jul 4 14:46:38 169.0.0.71 iscsi-ha: 17245 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Jul 4 14:46:38 169.0.0.71 iscsi-ha: 17245 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Jul 4 14:46:38 169.0.0.71 iscsi-ha: 17245 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:145459 dw:145459 dr:0 al:0 bm:27 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 Jul 4 14:46:38 169.0.0.71 iscsi-ha: 17245 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:145459 dw:145459 dr:0 al:0 bm:27 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 Jul 4 14:46:38 169.0.0.71 iscsi-ha: 17245 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Jul 4 14:46:38 169.0.0.71 iscsi-ha: 17245 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Jul 4 14:46:38 169.0.0.71 iscsi-ha: 17245 DRBD Resource: iscsi1 in WFConnection state - expected Connected state Jul 4 14:46:38 169.0.0.71 iscsi-ha: 17245 DRBD Resource: iscsi1 in WFConnection state - expected Connected state Jul 4 14:46:40 169.0.0.71 iscsid: connect to 10.10.10.3:3260 failed (No route to host) Jul 4 14:46:41 169.0.0.71 iscsid: connect to 10.10.10.3:3260 failed (No route to host) Jul 4 14:46:46 169.0.0.71 ha-lizard: 6957 ha-lizard Watchdog: ha-lizard running - OK Jul 4 14:46:46 169.0.0.71 ha-lizard: 6957 ha-lizard Watchdog: ha-lizard running - OK Jul 4 14:46:46 169.0.0.71 iscsi-ha: 7109 iscsi-ha Watchdog: iscsi-ha running - OK Jul 4 14:46:46 169.0.0.71 iscsi-ha: 7109 iscsi-ha Watchdog: iscsi-ha running - OK Jul 4 14:46:46 169.0.0.71 ha-lizard: 16816 ha-lizard already running: Attempt 4 on PIDS: 16824 16819 16818 16816 Jul 4 14:46:46 169.0.0.71 ha-lizard: 16816 ha-lizard already running: Attempt 4 on PIDS: 16824 16819 16818 16816 Jul 4 14:46:47 169.0.0.71 iscsid: connect to 10.10.10.3:3260 failed (No route to host) Jul 4 14:46:47 169.0.0.71 iscsi-ha: 17195 iscsi-ha already running: Attempt 1 on PIDS: 17203 17198 17197 Jul 4 14:46:47 169.0.0.71 iscsi-ha: 17195 iscsi-ha already running: Attempt 1 on PIDS: 17203 17198 17197 Jul 4 14:46:47 169.0.0.71 xapi: [ warn|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|xmlrpc_client] stunnel pid: 17220 caught Xmlrpc_client.Connection_reset Jul 4 14:46:47 169.0.0.71 xapi: [ warn|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|xmlrpc_client] stunnel pid: 17220 caught Xmlrpc_client.Connection_reset Jul 4 14:46:47 169.0.0.71 xapi: [error|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] Caught Unix.Unix_error(31, "write", "") Jul 4 14:46:47 169.0.0.71 xapi: [error|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] Caught Unix.Unix_error(31, "write", "") Jul 4 14:46:47 169.0.0.71 xapi: [error|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message). Jul 4 14:46:47 169.0.0.71 xapi: [error|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message). Jul 4 14:46:49 169.0.0.71 xapi: [ info|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] stunnel connected pid=17376 fd=22 Jul 4 14:46:49 169.0.0.71 xapi: [ info|XCP2|705 UNIX /var/xapi/xapi|session.login_with_password D:e5fcde6e3e26|master_connection] stunnel connected pid=17376 fd=22 Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17195 Spawning new instance of iscsi-ha Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17195 Spawning new instance of iscsi-ha Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17195 check_logger_processes Checking logger processes Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17195 check_logger_processes Checking logger processes Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17195 check_logger_processes No processes to clear Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17195 check_logger_processes No processes to clear Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17419 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17419 Mail Spool Directory Found /dev/shm/iscsi-ha-mail Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17419 This iteration is count 19 Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17419 This iteration is count 19 Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17419 Checking if this host is a Pool Master or Slave Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17419 Checking if this host is a Pool Master or Slave Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17419 This host's pool status = slave:169.0.0.70 Jul 4 14:46:52 169.0.0.71 iscsi-ha: 17419 This host's pool status = slave:169.0.0.70 Jul 4 14:46:53 169.0.0.71 iscsi-ha: 17453 local_ip_list: Local IP list returned 127.0.0.1 169.0.0.71 10.10.10.2 Jul 4 14:46:53 169.0.0.71 iscsi-ha: 17453 local_ip_list: Local IP list returned 127.0.0.1 169.0.0.71 10.10.10.2 Jul 4 14:46:53 169.0.0.71 iscsi-ha: 17453 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Jul 4 14:46:53 169.0.0.71 iscsi-ha: 17453 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped] Jul 4 14:46:53 169.0.0.71 iscsi-ha: 17453 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:145459 dw:145459 dr:0 al:0 bm:27 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 Jul 4 14:46:53 169.0.0.71 iscsi-ha: 17453 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r----- ns:0 nr:145459 dw:145459 dr:0 al:0 bm:27 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0 Jul 4 14:46:53 169.0.0.71 iscsi-ha: 17453 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Jul 4 14:46:53 169.0.0.71 iscsi-ha: 17453 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Jul 4 14:46:53 169.0.0.71 iscsi-ha: 17453 DRBD Resource: iscsi1 in WFConnection state - expected Connected state Jul 4 14:46:53 169.0.0.71 iscsi-ha: 17453 DRBD Resource: iscsi1 in WFConnection state - expected Connected state Jul 4 14:46:54 169.0.0.71 iscsid: connect to 10.10.10.3:3260 failed (No route to host) Jul 4 14:46:54 169.0.0.71 ha-lizard: 16824 check_xapi: Pool Host 169.0.0.70 failed to respond to ICMP ping Jul 4 14:46:54 169.0.0.71 ha-lizard: 16824 check_xapi: Pool Host 169.0.0.70 failed to respond to ICMP ping Jul 4 14:46:54 169.0.0.71 ha-lizard: 16824 Retrieving number of hosts in pool. Setting NUM_HOSTS = 1 Jul 4 14:46:54 169.0.0.71 ha-lizard: 16824 Retrieving number of hosts in pool. Setting NUM_HOSTS = 1 Jul 4 14:46:54 169.0.0.71 ha-lizard: 16824 Failed to reach Pool Master - Checking if this host promotes to Master.. Jul 4 14:46:54 169.0.0.71 ha-lizard: 16824 Failed to reach Pool Master - Checking if this host promotes to Master.. Jul 4 14:46:54 169.0.0.71 ha-lizard: 16824 State file MASTER UUID = c277f5d2-ce5f-46b6-859d-cf0a39ada1ca Jul 4 14:46:55 169.0.0.71 ha-lizard: 16824 State file MASTER UUID = c277f5d2-ce5f-46b6-859d-cf0a39ada1ca Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 fence_host: fence_host called to fencehost: c277f5d2-ce5f-46b6-859d-cf0a39ada1ca with action: stop and method: POOL. Check Pool! Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 fence_host: fence_host called to fencehost: c277f5d2-ce5f-46b6-859d-cf0a39ada1ca with action: stop and method: POOL. Check Pool! Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Checking host IPs: 169.0.0.71 Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Checking host IPs: 169.0.0.71 Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Host IP: 169.0.0.71 Response = OK Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Host IP: 169.0.0.71 Response = OK Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: LIVE HOSTs = 1 Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: LIVE HOSTs = 1 Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Using network points: 8.8.8.8 as possible additional vote Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Using network points: 8.8.8.8 as possible additional vote Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Heuristic IP: 8.8.8.8 Response = OK Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Heuristic IP: 8.8.8.8 Response = OK Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Successful Replies = 1 Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Successful Replies = 1 Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 Total enpoints checked = 1 with total successful replies = 1 Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 Total enpoints checked = 1 with total successful replies = 1 Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Minimum number of hosts needed to allow fencing = 0 + 1 Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: Minimum number of hosts needed to allow fencing = 0 + 1 Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: 2 Hosts found. Minimum needed = 0 + 1. Fencing allowed Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 check_quorum: 2 Hosts found. Minimum needed = 0 + 1. Fencing allowed Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 fence_host: Fencing is disabled or blocked. Check configuration or pool state. FENCE_ENABLED = 1, FENCE_BLOCK = , NUM_HOSTS = 1, FENCE_MIN_HOSTS = 2 - cannot fence host c277f5d2-ce5f-46b6-859d-cf0a39ada1ca Jul 4 14:46:55 169.0.0.71 ha-lizard: 17510 fence_host: Fencing is disabled or blocked. Check configuration or pool state. FENCE_ENABLED = 1, FENCE_BLOCK = , NUM_HOSTS = 1, FENCE_MIN_HOSTS = 2 - cannot fence host c277f5d2-ce5f-46b6-859d-cf0a39ada1ca Jul 4 14:46:55 169.0.0.71 ha-lizard: 16824 Function fence_host returned status 1 Jul 4 14:46:55 169.0.0.71 ha-lizard: 16824 Function fence_host returned status 1 Jul 4 14:46:55 169.0.0.71 ha-lizard: 16824 Failed to fence Master: c277f5d2-ce5f-46b6-859d-cf0a39ada1ca. Checking whether FENCE_HA_ONFAIL is enabled Jul 4 14:46:55 169.0.0.71 ha-lizard: 16824 Failed to fence Master: c277f5d2-ce5f-46b6-859d-cf0a39ada1ca. Checking whether FENCE_HA_ONFAIL is enabled Jul 4 14:46:55 169.0.0.71 ha-lizard: 16824 Marking this host as fenced, Rebooting this host now! Jul 4 14:46:55 169.0.0.71 ha-lizard: 16824 Marking this host as fenced, Rebooting this host now! Jul 4 14:46:55 169.0.0.71 ha-lizard: 16824 !!!!!!!!!!!!!!!!!!!!!!!!!!!!! SELF FENCING - REBOOT HERE !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Jul 4 14:46:55 169.0.0.71 ha-lizard: 16824 !!!!!!!!!!!!!!!!!!!!!!!!!!!!! SELF FENCING - REBOOT HERE !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!