Mar 26 09:49:47 xenserver-2646 syslogd 1.4.1: restart.
Mar 26 09:49:58 xenserver-2646 iscsi-ha: 8786 Spawning new instance of iscsi-ha
Mar 26 09:49:58 xenserver-2646 iscsi-ha: 8786 check_logger_processes Checking logger processes
Mar 26 09:49:58 xenserver-2646 iscsi-ha: 8786 check_logger_processes No processes to clear
Mar 26 09:49:58 xenserver-2646 iscsi-ha: 8817 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:49:58 xenserver-2646 iscsi-ha: 8817 This iteration is count 1
Mar 26 09:49:58 xenserver-2646 iscsi-ha: 8817 Checking if this host is a Pool Master or Slave
Mar 26 09:49:58 xenserver-2646 iscsi-ha: 8817 This host's pool status = slave:172.16.215.45
Mar 26 09:49:58 xenserver-2646 iscsi-ha: 8877 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:49:58 xenserver-2646 xapi: [ info|xenserver-2646|31725 UNIX /var/xapi/xapi|session.login_with_password D:4a7953c8ec18|xapi] Session.create trackid=2966d024d67ba6189282ad1b3e54f942 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:49:58 xenserver-2646 iscsi-ha: 8877 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:49:58 xenserver-2646 iscsi-ha: 8877 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----     ns:0 nr:362703568 dw:362703568 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:49:58 xenserver-2646 iscsi-ha: 8877 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:49:58 xenserver-2646 iscsi-ha: 8877 DRBD Resource: iscsi1 in Connected state
Mar 26 09:49:58 xenserver-2646 xapi: [ info|xenserver-2646|31726 UNIX /var/xapi/xapi|session.login_with_password D:7ce49ad18722|xapi] Session.create trackid=36d689074756a4b5c9a09aa5a005eca9 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:49:58 xenserver-2646 iscsi-ha-8810-INFO-/etc/iscsi-ha/iscsi-ha.sh: Scanning for Volume Group -> iscsi-sr: 65b6b083-7e57-da4c-84f9-b890cd24bf53
Mar 26 09:49:58 xenserver-2646 iscsi-ha-8810-INFO-/etc/iscsi-ha/iscsi-ha.sh: Volume Group for iSCSI-SR found OK: 65b6b083-7e57-da4c-84f9-b890cd24bf53
Mar 26 09:50:03 xenserver-2646 ha-lizard: 8999 Spawning new instance of ha-lizard
Mar 26 09:50:03 xenserver-2646 ha-lizard: 8999 check_logger_processes: Checking logger processes
Mar 26 09:50:03 xenserver-2646 ha-lizard: 8999 check_logger_processes: No processes to clear
Mar 26 09:50:03 xenserver-2646 ha-lizard: 9035 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Mar 26 09:50:03 xenserver-2646 ha-lizard: 9035 This iteration is count 1
Mar 26 09:50:03 xenserver-2646 ha-lizard: 9035 Checking if this host is a Pool Master or Slave
Mar 26 09:50:03 xenserver-2646 ha-lizard: 9035 This host's pool status = slave:172.16.215.45
Mar 26 09:50:03 xenserver-2646 xapi: [ info|xenserver-2646|31728 UNIX /var/xapi/xapi|session.login_with_password D:b2cc29a1b650|xapi] Session.create trackid=3827e72baff6570a41e60e63d8928e51 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:03 xenserver-2646 xapi: [ info|xenserver-2646|31729 UNIX /var/xapi/xapi|session.login_with_password D:e0cd7da61995|xapi] Session.create trackid=8952f5c9f0d9bceeff44be5b39b520dc pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:03 xenserver-2646 ha-lizard: 9035 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run
Mar 26 09:50:03 xenserver-2646 ha-lizard: 9035 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=0 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=1 FENCE_HEURISTICS_IPS=() FENCE_HOST_FORGET=0 FENCE_IPADDRESS=8.8.8.8\,4.2.2.2 FENCE_METHOD=POOL FENCE_MIN_HOSTS=3 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=0 MAIL_FROM=root@xenserver-metaphors MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO=email@domain.net MONITOR_DELAY=45 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=50 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=5 XAPI_DELAY=15 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10
Mar 26 09:50:03 xenserver-2646 ha-lizard: 9035 master_ip: Pool Master IP Address =  172.16.215.45
Mar 26 09:50:03 xenserver-2646 ha-lizard: 9035 This slave - xenserver-2646: 9a7810fe-29d1-4c21-adc6-b4aeb003e874 selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Mar 26 09:50:03 xenserver-2646 ha-lizard: 9035 check_xapi: Pool Host 172.16.215.45 xapi status = 1
Mar 26 09:50:03 xenserver-2646 ha-lizard: 9035 Pool Master is OK - calling function check_ha_enabled - updating local status
Mar 26 09:50:03 xenserver-2646 xapi: [ info|xenserver-2646|31730 UNIX /var/xapi/xapi|session.login_with_password D:16514452df99|xapi] Session.create trackid=4fea0e15e8394cf2653d26882b5c2ac0 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:03 xenserver-2646 xapi: [ info|xenserver-2646|31731 UNIX /var/xapi/xapi|session.login_with_password D:19fdbdc9198a|xapi] Session.create trackid=7816c618bb538527d958c3c9bdc50243 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:04 xenserver-2646 ha-lizard: 9035 check_ha_enabled: Checking if ha-lizard is enabled for pool: 0b64d0d8-f133-0b22-29f8-b6a34c7cdd6a
Mar 26 09:50:04 xenserver-2646 ha-lizard: 9035 check_ha_enabled: ha-lizard is enabled
Mar 26 09:50:04 xenserver-2646 ha-lizard: 9035 Checking state file for status if ha-lizard is enabled
Mar 26 09:50:04 xenserver-2646 ha-lizard: 9035 Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Mar 26 09:50:04 xenserver-2646 ha-lizard: 9035 ha-lizard is enabled - continuing
Mar 26 09:50:04 xenserver-2646 ha-lizard: 9035 Calling Function write_pool_state - updating local state files
Mar 26 09:50:04 xenserver-2646 xapi: [ info|xenserver-2646|31732 UNIX /var/xapi/xapi|session.login_with_password D:3bca1fd675c9|xapi] Session.create trackid=576029dd096e9f822bbda4bfad339ea3 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:04 xenserver-2646 xapi: [ info|xenserver-2646|31733 UNIX /var/xapi/xapi|session.login_with_password D:77ae2329337f|xapi] Session.create trackid=b80b57f6c01abc8c97e8d7a9a629b49a pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:04 xenserver-2646 xapi: [ info|xenserver-2646|31734 UNIX /var/xapi/xapi|session.login_with_password D:e9cdd7955429|xapi] Session.create trackid=5a5eb2089a69601ca088554245a6e2c8 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:04 xenserver-2646 ha-lizard: 9035 get_vms_on_host: No VMs found on host: 9a7810fe-29d1-4c21-adc6-b4aeb003e874
Mar 26 09:50:04 xenserver-2646 xapi: [ info|xenserver-2646|31735 UNIX /var/xapi/xapi|session.login_with_password D:3d6a0bfa8561|xapi] Session.create trackid=fa6e4d09ba583ca0518484844e155417 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:04 xenserver-2646 ha-lizard: 9035 get_vms_on_host: Returned 463f3569-29c9-938a-a1f0-caeae6e47750
Mar 26 09:50:04 xenserver-2646 xapi: [ info|xenserver-2646|31736 UNIX /var/xapi/xapi|session.login_with_password D:3734b7e6f2a7|xapi] Session.create trackid=99f6b8d6d541aefdab985afe9e9accd4 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:04 xenserver-2646 xapi: [ info|xenserver-2646|31737 UNIX /var/xapi/xapi|session.login_with_password D:3a888cc7aaa6|xapi] Session.create trackid=92bc6da8a773995de9471dc7d9baf025 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:04 xenserver-2646 xapi: [ info|xenserver-2646|31738 UNIX /var/xapi/xapi|session.login_with_password D:6d229a04c7b8|xapi] Session.create trackid=bc177402a775210c6441b8a16b080381 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:04 xenserver-2646 xapi: [ info|xenserver-2646|31739 UNIX /var/xapi/xapi|session.login_with_password D:66a49dee6321|xapi] Session.create trackid=c3acfb72787082a4057adc61b90e2a2f pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:04 xenserver-2646 ha-lizard: 9035 check_ha_enabled: Checking if ha-lizard is enabled for pool: 0b64d0d8-f133-0b22-29f8-b6a34c7cdd6a
Mar 26 09:50:04 xenserver-2646 ha-lizard: 9035 check_ha_enabled: ha-lizard is enabled
Mar 26 09:50:04 xenserver-2646 xapi: [ info|xenserver-2646|31740 UNIX /var/xapi/xapi|session.login_with_password D:a693c78dc73e|xapi] Session.create trackid=cbed20d86012ddfc83a3a2adc092d4f2 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:04 xenserver-2646 xapi: [ info|xenserver-2646|31741 UNIX /var/xapi/xapi|session.login_with_password D:04e77b67e44d|xapi] Session.create trackid=fc4160b77ba085bf42f4072d8740f2aa pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:04 xenserver-2646 ha-lizard: 9035 get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Mar 26 09:50:05 xenserver-2646 xapi: [ info|xenserver-2646|31742 UNIX /var/xapi/xapi|session.login_with_password D:abdcc07f8579|xapi] Session.create trackid=a52b3319ab7e2a7acaa483cd251dba33 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:05 xenserver-2646 ha-lizard: 9035 get_pool_host_list: returned 9a7810fe-29d1-4c21-adc6-b4aeb003e874 280a9657-763b-4909-a740-97fc63b4a51d
Mar 26 09:50:05 xenserver-2646 xapi: [ info|xenserver-2646|31743 UNIX /var/xapi/xapi|session.login_with_password D:bdc28654eaac|xapi] Session.create trackid=aa7ee89c4fff70f340138ea8b0ba4ae7 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:05 xenserver-2646 ha-lizard: 9035 get_pool_ip_list: returned 172.16.215.46
Mar 26 09:50:05 xenserver-2646 xapi: [ info|xenserver-2646|31744 UNIX /var/xapi/xapi|session.login_with_password D:046a2643679b|xapi] Session.create trackid=39fab2e5f6abb768fd88ccc6e78ff3eb pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:05 xenserver-2646 ha-lizard: 9035 get_pool_ip_list: returned 172.16.215.46 172.16.215.45
Mar 26 09:50:08 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:50:08 xenserver-2646 iscsi-ha: 8807 Spawning new instance of iscsi-ha
Mar 26 09:50:08 xenserver-2646 iscsi-ha: 8807 check_logger_processes Checking logger processes
Mar 26 09:50:08 xenserver-2646 iscsi-ha: 8807 check_logger_processes No processes to clear
Mar 26 09:50:08 xenserver-2646 iscsi-ha: 9345 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:50:08 xenserver-2646 iscsi-ha: 9345 This iteration is count 2
Mar 26 09:50:08 xenserver-2646 iscsi-ha: 9345 Checking if this host is a Pool Master or Slave
Mar 26 09:50:08 xenserver-2646 iscsi-ha: 9345 This host's pool status = slave:172.16.215.45
Mar 26 09:50:08 xenserver-2646 iscsi-ha: 9380 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:50:08 xenserver-2646 xapi: [ info|xenserver-2646|31745 UNIX /var/xapi/xapi|session.login_with_password D:47c28e462aa4|xapi] Session.create trackid=4facd41225949143a1c31a300b620b7a pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:08 xenserver-2646 iscsi-ha: 9380 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:50:08 xenserver-2646 iscsi-ha: 9380 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----     ns:0 nr:362703568 dw:362703568 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:50:08 xenserver-2646 iscsi-ha: 9380 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:50:08 xenserver-2646 iscsi-ha: 9380 DRBD Resource: iscsi1 in Connected state
Mar 26 09:50:08 xenserver-2646 xapi: [ info|xenserver-2646|31746 UNIX /var/xapi/xapi|session.login_with_password D:966b2a8510ba|xapi] Session.create trackid=5a73d417440196ed61666effb915deb1 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:08 xenserver-2646 iscsi-ha-9340-INFO-/etc/iscsi-ha/iscsi-ha.sh: Scanning for Volume Group -> iscsi-sr: 65b6b083-7e57-da4c-84f9-b890cd24bf53
Mar 26 09:50:08 xenserver-2646 iscsi-ha-9340-INFO-/etc/iscsi-ha/iscsi-ha.sh: Volume Group for iSCSI-SR found OK: 65b6b083-7e57-da4c-84f9-b890cd24bf53
Mar 26 09:50:13 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:50:18 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:50:18 xenserver-2646 iscsi-ha: 9337 Spawning new instance of iscsi-ha
Mar 26 09:50:18 xenserver-2646 iscsi-ha: 9337 check_logger_processes Checking logger processes
Mar 26 09:50:18 xenserver-2646 iscsi-ha: 9337 check_logger_processes No processes to clear
Mar 26 09:50:18 xenserver-2646 iscsi-ha: 9468 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:50:18 xenserver-2646 iscsi-ha: 9468 This iteration is count 3
Mar 26 09:50:18 xenserver-2646 iscsi-ha: 9468 Checking if this host is a Pool Master or Slave
Mar 26 09:50:18 xenserver-2646 iscsi-ha: 9468 This host's pool status = slave:172.16.215.45
Mar 26 09:50:18 xenserver-2646 iscsi-ha: 9503 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:50:18 xenserver-2646 xapi: [ info|xenserver-2646|31747 UNIX /var/xapi/xapi|session.login_with_password D:0e5737960cfd|xapi] Session.create trackid=78025d20126265f615dd850ddf449ae4 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:18 xenserver-2646 iscsi-ha: 9503 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:50:18 xenserver-2646 iscsi-ha: 9503 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----     ns:0 nr:362703568 dw:362703568 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:50:18 xenserver-2646 iscsi-ha: 9503 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:50:18 xenserver-2646 iscsi-ha: 9503 DRBD Resource: iscsi1 in Connected state
Mar 26 09:50:18 xenserver-2646 xapi: [ info|xenserver-2646|31748 UNIX /var/xapi/xapi|session.login_with_password D:0100a2f8ed2b|xapi] Session.create trackid=29fe898159a6c2db9e4f41df430473d1 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:18 xenserver-2646 iscsi-ha-9463-INFO-/etc/iscsi-ha/iscsi-ha.sh: Scanning for Volume Group -> iscsi-sr: 65b6b083-7e57-da4c-84f9-b890cd24bf53
Mar 26 09:50:18 xenserver-2646 iscsi-ha-9463-INFO-/etc/iscsi-ha/iscsi-ha.sh: Volume Group for iSCSI-SR found OK: 65b6b083-7e57-da4c-84f9-b890cd24bf53
Mar 26 09:50:23 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:50:28 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:50:28 xenserver-2646 iscsi-ha: 9460 Spawning new instance of iscsi-ha
Mar 26 09:50:28 xenserver-2646 iscsi-ha: 9460 check_logger_processes Checking logger processes
Mar 26 09:50:28 xenserver-2646 iscsi-ha: 9460 check_logger_processes No processes to clear
Mar 26 09:50:28 xenserver-2646 iscsi-ha: 9582 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:50:28 xenserver-2646 iscsi-ha: 9582 This iteration is count 4
Mar 26 09:50:28 xenserver-2646 iscsi-ha: 9582 Checking if this host is a Pool Master or Slave
Mar 26 09:50:28 xenserver-2646 iscsi-ha: 9582 This host's pool status = slave:172.16.215.45
Mar 26 09:50:28 xenserver-2646 iscsi-ha: 9617 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:50:28 xenserver-2646 xapi: [ info|xenserver-2646|31749 UNIX /var/xapi/xapi|session.login_with_password D:4c8d04e23c53|xapi] Session.create trackid=3d6b7e1bad9a8a93f9db033d568f59ae pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:28 xenserver-2646 iscsi-ha: 9617 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:50:28 xenserver-2646 iscsi-ha: 9617 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----     ns:0 nr:362706352 dw:362706352 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:50:28 xenserver-2646 iscsi-ha: 9617 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:50:28 xenserver-2646 iscsi-ha: 9617 DRBD Resource: iscsi1 in Connected state
Mar 26 09:50:28 xenserver-2646 xapi: [ info|xenserver-2646|31750 UNIX /var/xapi/xapi|session.login_with_password D:dc122eb35649|xapi] Session.create trackid=7649505b43a2858ef648cd119eb52f0e pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:28 xenserver-2646 iscsi-ha-9577-INFO-/etc/iscsi-ha/iscsi-ha.sh: Scanning for Volume Group -> iscsi-sr: 65b6b083-7e57-da4c-84f9-b890cd24bf53
Mar 26 09:50:28 xenserver-2646 iscsi-ha-9577-INFO-/etc/iscsi-ha/iscsi-ha.sh: Volume Group for iSCSI-SR found OK: 65b6b083-7e57-da4c-84f9-b890cd24bf53
Mar 26 09:50:33 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:50:38 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:50:38 xenserver-2646 iscsi-ha: 9574 Spawning new instance of iscsi-ha
Mar 26 09:50:38 xenserver-2646 iscsi-ha: 9574 check_logger_processes Checking logger processes
Mar 26 09:50:38 xenserver-2646 iscsi-ha: 9574 check_logger_processes No processes to clear
Mar 26 09:50:38 xenserver-2646 iscsi-ha: 9700 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:50:38 xenserver-2646 iscsi-ha: 9700 This iteration is count 5
Mar 26 09:50:38 xenserver-2646 iscsi-ha: 9700 Checking if this host is a Pool Master or Slave
Mar 26 09:50:38 xenserver-2646 iscsi-ha: 9700 This host's pool status = slave:172.16.215.45
Mar 26 09:50:38 xenserver-2646 iscsi-ha: 9735 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:50:38 xenserver-2646 xapi: [ info|xenserver-2646|31751 UNIX /var/xapi/xapi|session.login_with_password D:7d0aa4a05e9e|xapi] Session.create trackid=599c96a156aa5450cf2023dc340d2778 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:38 xenserver-2646 iscsi-ha: 9735 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:50:38 xenserver-2646 iscsi-ha: 9735 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----     ns:0 nr:362706352 dw:362706352 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:50:38 xenserver-2646 iscsi-ha: 9735 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:50:38 xenserver-2646 iscsi-ha: 9735 DRBD Resource: iscsi1 in Connected state
Mar 26 09:50:38 xenserver-2646 xapi: [ info|xenserver-2646|31752 UNIX /var/xapi/xapi|session.login_with_password D:8e7b71bee3b7|xapi] Session.create trackid=5db8ee9d2610e873524166614fcb5079 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:38 xenserver-2646 iscsi-ha-9695-INFO-/etc/iscsi-ha/iscsi-ha.sh: Scanning for Volume Group -> iscsi-sr: 65b6b083-7e57-da4c-84f9-b890cd24bf53
Mar 26 09:50:38 xenserver-2646 iscsi-ha-9695-INFO-/etc/iscsi-ha/iscsi-ha.sh: Volume Group for iSCSI-SR found OK: 65b6b083-7e57-da4c-84f9-b890cd24bf53
Mar 26 09:50:43 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:50:48 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:50:48 xenserver-2646 iscsi-ha: 9692 Spawning new instance of iscsi-ha
Mar 26 09:50:48 xenserver-2646 iscsi-ha: 9692 check_logger_processes Checking logger processes
Mar 26 09:50:48 xenserver-2646 iscsi-ha: 9692 check_logger_processes No processes to clear
Mar 26 09:50:48 xenserver-2646 iscsi-ha: 9815 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:50:48 xenserver-2646 iscsi-ha: 9815 This iteration is count 6
Mar 26 09:50:48 xenserver-2646 iscsi-ha: 9815 Checking if this host is a Pool Master or Slave
Mar 26 09:50:48 xenserver-2646 iscsi-ha: 9815 This host's pool status = slave:172.16.215.45
Mar 26 09:50:48 xenserver-2646 iscsi-ha: 9850 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:50:48 xenserver-2646 xapi: [ info|xenserver-2646|31754 UNIX /var/xapi/xapi|session.login_with_password D:4d0d487ac537|xapi] Session.create trackid=6b0a28b4d3076edb4c6b60ddb3f0352a pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:48 xenserver-2646 ha-lizard: 9026 Spawning new instance of ha-lizard
Mar 26 09:50:48 xenserver-2646 ha-lizard: 9026 check_logger_processes: Checking logger processes
Mar 26 09:50:48 xenserver-2646 ha-lizard: 9026 check_logger_processes: No processes to clear
Mar 26 09:50:48 xenserver-2646 iscsi-ha: 9850 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:50:48 xenserver-2646 iscsi-ha: 9850 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----     ns:0 nr:362706352 dw:362706352 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:50:48 xenserver-2646 iscsi-ha: 9850 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:50:48 xenserver-2646 iscsi-ha: 9850 DRBD Resource: iscsi1 in Connected state
Mar 26 09:50:48 xenserver-2646 ha-lizard: 9907 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Mar 26 09:50:48 xenserver-2646 ha-lizard: 9907 This iteration is count 2
Mar 26 09:50:48 xenserver-2646 ha-lizard: 9907 Checking if this host is a Pool Master or Slave
Mar 26 09:50:48 xenserver-2646 ha-lizard: 9907 This host's pool status = slave:172.16.215.45
Mar 26 09:50:48 xenserver-2646 xapi: [ info|xenserver-2646|31755 UNIX /var/xapi/xapi|session.login_with_password D:5a2b478cadf3|xapi] Session.create trackid=d7cb068c570096678af17ca701e48dbf pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:48 xenserver-2646 xapi: [ info|xenserver-2646|31756 UNIX /var/xapi/xapi|session.login_with_password D:f9d64a8a1492|xapi] Session.create trackid=c52322913cda4db511f8b9ee6b8b1b59 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:48 xenserver-2646 xapi: [ info|xenserver-2646|31757 UNIX /var/xapi/xapi|session.login_with_password D:6b285fcd774c|xapi] Session.create trackid=0f0a42e6c991d1072aa4a28fbe165b0d pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:48 xenserver-2646 iscsi-ha-9810-INFO-/etc/iscsi-ha/iscsi-ha.sh: Scanning for Volume Group -> iscsi-sr: 65b6b083-7e57-da4c-84f9-b890cd24bf53
Mar 26 09:50:48 xenserver-2646 iscsi-ha-9810-INFO-/etc/iscsi-ha/iscsi-ha.sh: Volume Group for iSCSI-SR found OK: 65b6b083-7e57-da4c-84f9-b890cd24bf53
Mar 26 09:50:48 xenserver-2646 ha-lizard: 9907 Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Settings take effect on subsequent run
Mar 26 09:50:48 xenserver-2646 ha-lizard: 9907 DISABLED_VAPPS=() ENABLE_LOGGING=1 FENCE_ACTION=stop FENCE_ENABLED=0 FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_HA_ONFAIL=1 FENCE_HEURISTICS_IPS=() FENCE_HOST_FORGET=0 FENCE_IPADDRESS=8.8.8.8\,4.2.2.2 FENCE_METHOD=POOL FENCE_MIN_HOSTS=3 FENCE_PASSWD= FENCE_QUORUM_REQUIRED=1 FENCE_REBOOT_LONE_HOST=0 FENCE_USE_IP_HEURISTICS=1 GLOBAL_VM_HA=0 MAIL_FROM=root@xenserver-metaphors MAIL_ON=1 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" MAIL_TO=email@domain.net MONITOR_DELAY=45 MONITOR_KILLALL=1 MONITOR_MAX_STARTS=50 MONITOR_SCANRATE=10 OP_MODE=2 PROMOTE_SLAVE=1 SLAVE_HA=1 SLAVE_VM_STAT=0 XAPI_COUNT=5 XAPI_DELAY=15 XC_FIELD_NAME='ha-lizard-enabled' XE_TIMEOUT=10
Mar 26 09:50:48 xenserver-2646 ha-lizard: 9907 master_ip: Pool Master IP Address =  172.16.215.45
Mar 26 09:50:48 xenserver-2646 ha-lizard: 9907 This slave - xenserver-2646: 9a7810fe-29d1-4c21-adc6-b4aeb003e874 selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Mar 26 09:50:48 xenserver-2646 ha-lizard: 9907 check_xapi: Pool Host 172.16.215.45 xapi status = 1
Mar 26 09:50:48 xenserver-2646 ha-lizard: 9907 Pool Master is OK - calling function check_ha_enabled - updating local status
Mar 26 09:50:48 xenserver-2646 xapi: [ info|xenserver-2646|31758 UNIX /var/xapi/xapi|session.login_with_password D:81dcd2725a63|xapi] Session.create trackid=892e0b257d40f0d21ba667a7ed9e8601 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:48 xenserver-2646 xapi: [ info|xenserver-2646|31759 UNIX /var/xapi/xapi|session.login_with_password D:d92520823a00|xapi] Session.create trackid=acd8ae8ddd0de7b8f0e47d23d40d2b64 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:49 xenserver-2646 ha-lizard: 9907 check_ha_enabled: Checking if ha-lizard is enabled for pool: 0b64d0d8-f133-0b22-29f8-b6a34c7cdd6a
Mar 26 09:50:49 xenserver-2646 ha-lizard: 9907 check_ha_enabled: ha-lizard is enabled
Mar 26 09:50:49 xenserver-2646 ha-lizard: 9907 Checking state file for status if ha-lizard is enabled
Mar 26 09:50:49 xenserver-2646 ha-lizard: 9907 Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Mar 26 09:50:49 xenserver-2646 ha-lizard: 9907 ha-lizard is enabled - continuing
Mar 26 09:50:49 xenserver-2646 ha-lizard: 9907 Calling Function write_pool_state - updating local state files
Mar 26 09:50:49 xenserver-2646 xapi: [ info|xenserver-2646|31760 UNIX /var/xapi/xapi|session.login_with_password D:00a5cec4b7db|xapi] Session.create trackid=99f75f5fac1e4f0fbfba68d38a1813b5 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:49 xenserver-2646 xapi: [ info|xenserver-2646|31761 UNIX /var/xapi/xapi|session.login_with_password D:b41dd4409b19|xapi] Session.create trackid=b5d21483a1e2e5d3ac7abd4eb912d7dc pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:49 xenserver-2646 xapi: [ info|xenserver-2646|31762 UNIX /var/xapi/xapi|session.login_with_password D:b08d5b6b8234|xapi] Session.create trackid=a21466d89d1028c85956a53e1b32ae15 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:49 xenserver-2646 ha-lizard: 9907 get_vms_on_host: No VMs found on host: 9a7810fe-29d1-4c21-adc6-b4aeb003e874
Mar 26 09:50:49 xenserver-2646 xapi: [ info|xenserver-2646|31763 UNIX /var/xapi/xapi|session.login_with_password D:3de1ec9e8259|xapi] Session.create trackid=303cbdc2edcdb3d706e725237690ae72 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:49 xenserver-2646 ha-lizard: 9907 get_vms_on_host: Returned 463f3569-29c9-938a-a1f0-caeae6e47750
Mar 26 09:50:49 xenserver-2646 xapi: [ info|xenserver-2646|31764 UNIX /var/xapi/xapi|session.login_with_password D:7f5b051ce045|xapi] Session.create trackid=44cc32e1e08ac56292514936b07aa05e pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:49 xenserver-2646 xapi: [ info|xenserver-2646|31765 UNIX /var/xapi/xapi|session.login_with_password D:aa990dec7ef3|xapi] Session.create trackid=177e0dada869dec0bdf8a57351fba143 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:49 xenserver-2646 xapi: [ info|xenserver-2646|31766 UNIX /var/xapi/xapi|session.login_with_password D:509a4a4d1524|xapi] Session.create trackid=2dcd14c4950984d9dc7a9b0c2de97194 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:49 xenserver-2646 xapi: [ info|xenserver-2646|31767 UNIX /var/xapi/xapi|session.login_with_password D:f39b063aeb63|xapi] Session.create trackid=2b4395bb6df63e7cfc178a0815ded24b pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:49 xenserver-2646 ha-lizard: 9907 check_ha_enabled: Checking if ha-lizard is enabled for pool: 0b64d0d8-f133-0b22-29f8-b6a34c7cdd6a
Mar 26 09:50:49 xenserver-2646 ha-lizard: 9907 check_ha_enabled: ha-lizard is enabled
Mar 26 09:50:49 xenserver-2646 xapi: [ info|xenserver-2646|31768 UNIX /var/xapi/xapi|session.login_with_password D:b001ef0b6618|xapi] Session.create trackid=5b28a7b743e9bcd40cebd7212e107591 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:49 xenserver-2646 xapi: [ info|xenserver-2646|31769 UNIX /var/xapi/xapi|session.login_with_password D:441c3370abea|xapi] Session.create trackid=be32608d765a2e909fdbbe13f5ea64c3 pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Mar 26 09:50:50 xenserver-2646 kernel: [53582.762448] igb: eth1 NIC Link is Down
Mar 26 09:50:53 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:50:54 xenserver-2646 kernel: [53586.503164] igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Mar 26 09:50:55 xenserver-2646 kernel: [53587.152080] block drbd1: PingAck did not arrive in time.
Mar 26 09:50:55 xenserver-2646 kernel: [53587.152103] block drbd1: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
Mar 26 09:50:55 xenserver-2646 kernel: [53587.152276] block drbd1: asender terminated
Mar 26 09:50:55 xenserver-2646 kernel: [53587.152284] block drbd1: Terminating drbd1_asender
Mar 26 09:50:55 xenserver-2646 kernel: [53587.152534] block drbd1: Connection closed
Mar 26 09:50:55 xenserver-2646 kernel: [53587.152543] block drbd1: conn( NetworkFailure -> Unconnected ) 
Mar 26 09:50:55 xenserver-2646 kernel: [53587.152554] block drbd1: receiver terminated
Mar 26 09:50:55 xenserver-2646 kernel: [53587.152558] block drbd1: Restarting drbd1_receiver
Mar 26 09:50:55 xenserver-2646 kernel: [53587.152562] block drbd1: receiver (re)started
Mar 26 09:50:55 xenserver-2646 kernel: [53587.152569] block drbd1: conn( Unconnected -> WFConnection ) 
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 9807 Spawning new instance of iscsi-ha
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 9807 check_logger_processes Checking logger processes
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 9807 check_logger_processes No processes to clear
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 10152 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 10152 This iteration is count 7
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 10152 Checking if this host is a Pool Master or Slave
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 10152 This host's pool status = slave:172.16.215.45
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 10187 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 10187 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 10187 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----     ns:0 nr:362706352 dw:362706352 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 10187 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 10187 DRBD Resource: iscsi1 in WFConnection state - expected Connected state
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 10187 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:50:58 xenserver-2646 iscsi-ha-NOTICE-/etc/iscsi-ha/iscsi-ha.sh: cat: /dev/shm/iscsi-ha-mail/*.msg: No such file or directory
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 10187 email Sending ALERT email to root@localhost: DRBD Resource:       iscsi1 in WFConnection state - expected Connected state
Mar 26 09:50:58 xenserver-2646 iscsi-ha-10147-INFO-/etc/iscsi-ha/iscsi-ha.sh: Sending email from: root@localhost
Mar 26 09:50:58 xenserver-2646 iscsi-ha-10147-INFO-/etc/iscsi-ha/iscsi-ha.sh: Sending email to: root@localhost
Mar 26 09:50:58 xenserver-2646 iscsi-ha-10147-INFO-/etc/iscsi-ha/iscsi-ha.sh: Email Alert Subject: HA-Lizard noSAN SYSTEM ALERT - FROM HOST: xenserver-2646
Mar 26 09:50:58 xenserver-2646 iscsi-ha-10147-INFO-/etc/iscsi-ha/iscsi-ha.sh: Email Alert Timestamp: Wed Mar 26 09:50:58 GMT 2014
Mar 26 09:50:58 xenserver-2646 iscsi-ha-10147-INFO-/etc/iscsi-ha/iscsi-ha.sh: Email Alert Process: iscsi-ha Version: IHA_1.3.7
Mar 26 09:50:58 xenserver-2646 iscsi-ha-10147-INFO-/etc/iscsi-ha/iscsi-ha.sh: Email Alert Message Content: DRBD Resource:       iscsi1 in WFConnection state - expected Connected state
Mar 26 09:50:58 xenserver-2646 iscsi-ha-10147-INFO-/etc/iscsi-ha/iscsi-ha.sh: Email Alert Message Hostname: xenserver-2646
Mar 26 09:50:58 xenserver-2646 iscsi-ha: 10187 email: Message copied to /dev/shm/iscsi-ha-mail/1395827458.msg - Suppressing duplicates for 30 Minutes
Mar 26 09:50:59 xenserver-2646 ha-lizard: 9907 xe_wrapper: COMMAND: xe host-list enabled=true params=name-label Has reached the maximum allowable time of 10 seconds. Killing all processes now!
Mar 26 09:51:03 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:51:08 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:51:08 xenserver-2646 iscsi-ha: 10144 iscsi-ha already running: Attempt 1 on PIDS: 10152 10147 10146
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10144 Spawning new instance of iscsi-ha
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10144 check_logger_processes Checking logger processes
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10144 check_logger_processes No processes to clear
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10325 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10325 This iteration is count 8
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10325 Checking if this host is a Pool Master or Slave
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10325 This host's pool status = slave:172.16.215.45
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10362 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:51:13 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10362 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10362 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----     ns:0 nr:362706352 dw:362706352 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:51:13 xenserver-2646 kernel: [53605.808705]  connection1:0: ping timeout of 15 secs expired, recv timeout 10, last rx 5327805, last ping 5328805, now 5330305
Mar 26 09:51:13 xenserver-2646 kernel: [53605.808713]  connection1:0: detected conn error (1011)
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10362 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10362 DRBD Resource: iscsi1 in WFConnection state - expected Connected state
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10362 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10362 email: Duplicate message - not sending. Content = DRBD Resource:       iscsi1 in WFConnection state - expected Connected state
Mar 26 09:51:13 xenserver-2646 iscsi-ha: 10362 email: Message barred for 30 minutes
Mar 26 09:51:14 xenserver-2646 iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3)
Mar 26 09:51:18 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:51:23 xenserver-2646 iscsi-ha: 10317 iscsi-ha already running: Attempt 1 on PIDS: 10325 10320 10319
Mar 26 09:51:23 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10317 Spawning new instance of iscsi-ha
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10317 check_logger_processes Checking logger processes
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10317 check_logger_processes No processes to clear
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10458 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10458 This iteration is count 9
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10458 Checking if this host is a Pool Master or Slave
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10458 This host's pool status = slave:172.16.215.45
Mar 26 09:51:28 xenserver-2646 iscsid: connect to 192.168.215.44:3260 failed (No route to host) 
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10493 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10493 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10493 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----     ns:0 nr:362706352 dw:362706352 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10493 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10493 DRBD Resource: iscsi1 in WFConnection state - expected Connected state
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10493 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10493 email: Duplicate message - not sending. Content = DRBD Resource:       iscsi1 in WFConnection state - expected Connected state
Mar 26 09:51:28 xenserver-2646 iscsi-ha: 10493 email: Message barred for 30 minutes
Mar 26 09:51:31 xenserver-2646 iscsi-ha-10453-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 172.16.215.45 not responding - replug_pbd exiting
Mar 26 09:51:33 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard is stopped, starting ha-lizard
Mar 26 09:51:33 xenserver-2646 ha-lizard: 10571 Spawning new instance of ha-lizard
Mar 26 09:51:33 xenserver-2646 ha-lizard: 10571 check_logger_processes: Checking logger processes
Mar 26 09:51:34 xenserver-2646 ha-lizard: 10571 check_logger_processes: No processes to clear
Mar 26 09:51:34 xenserver-2646 ha-lizard: 10603 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Mar 26 09:51:34 xenserver-2646 ha-lizard: 10603 This iteration is count 1
Mar 26 09:51:34 xenserver-2646 ha-lizard: 10603 Checking if this host is a Pool Master or Slave
Mar 26 09:51:34 xenserver-2646 ha-lizard: 10603 This host's pool status = slave:172.16.215.45
Mar 26 09:51:34 xenserver-2646 iscsid: connect to 192.168.215.44:3260 failed (No route to host) 
Mar 26 09:51:37 xenserver-2646 ha-lizard: 10603 Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration!
Mar 26 09:51:37 xenserver-2646 ha-lizard: 10603 master_ip: Pool Master IP Address =  172.16.215.45
Mar 26 09:51:37 xenserver-2646 ha-lizard: 10603 This slave - xenserver-2646: 9a7810fe-29d1-4c21-adc6-b4aeb003e874 selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10450 Spawning new instance of iscsi-ha
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10450 check_logger_processes Checking logger processes
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10450 check_logger_processes No processes to clear
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10650 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10650 This iteration is count 10
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10650 Checking if this host is a Pool Master or Slave
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10650 This host's pool status = slave:172.16.215.45
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10685 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10685 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10685 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----     ns:0 nr:362706352 dw:362706352 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10685 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10685 DRBD Resource: iscsi1 in WFConnection state - expected Connected state
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10685 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10685 email: Duplicate message - not sending. Content = DRBD Resource:       iscsi1 in WFConnection state - expected Connected state
Mar 26 09:51:38 xenserver-2646 iscsi-ha: 10685 email: Message barred for 30 minutes
Mar 26 09:51:40 xenserver-2646 iscsi-ha-10645-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 172.16.215.45 not responding - replug_pbd exiting
Mar 26 09:51:40 xenserver-2646 ha-lizard: 10603 check_xapi: Pool Host 172.16.215.45 failed to respond to ICMP ping
Mar 26 09:51:40 xenserver-2646 ha-lizard: 10603 Pool Master NOT OK - Checking if ha-lizard is enabled in latest state file
Mar 26 09:51:40 xenserver-2646 ha-lizard: 10603 Checking if ha-lizard is enabled
Mar 26 09:51:40 xenserver-2646 ha-lizard: 10603 Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Mar 26 09:51:40 xenserver-2646 ha-lizard: 10603 ha-lizard is enabled - continuing
Mar 26 09:51:40 xenserver-2646 ha-lizard: 10603 Pool Master Monitor = Failed
Mar 26 09:51:40 xenserver-2646 ha-lizard: 10603 Retry Count set to 5. Retrying 5 times in 15 second intervals..
Mar 26 09:51:40 xenserver-2646 ha-lizard: 10603 Attempt 0: Checking Pool Master Status
Mar 26 09:51:40 xenserver-2646 iscsid: connect to 192.168.215.44:3260 failed (No route to host) 
Mar 26 09:51:44 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:51:48 xenserver-2646 iscsid: connect to 192.168.215.44:3260 failed (No route to host) 
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10642 Spawning new instance of iscsi-ha
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10642 check_logger_processes Checking logger processes
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10642 check_logger_processes No processes to clear
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10772 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10772 This iteration is count 11
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10772 Checking if this host is a Pool Master or Slave
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10772 This host's pool status = slave:172.16.215.45
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10807 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10807 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10807 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----     ns:0 nr:362706352 dw:362706352 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10807 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10807 DRBD Resource: iscsi1 in WFConnection state - expected Connected state
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10807 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10807 email: Duplicate message - not sending. Content = DRBD Resource:       iscsi1 in WFConnection state - expected Connected state
Mar 26 09:51:48 xenserver-2646 iscsi-ha: 10807 email: Message barred for 30 minutes
Mar 26 09:51:51 xenserver-2646 iscsi-ha-10767-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 172.16.215.45 not responding - replug_pbd exiting
Mar 26 09:51:54 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:51:54 xenserver-2646 iscsid: connect to 192.168.215.44:3260 failed (No route to host) 
Mar 26 09:51:58 xenserver-2646 ha-lizard: 10603 check_xapi: Pool Host 172.16.215.45 failed to respond to ICMP ping
Mar 26 09:51:58 xenserver-2646 ha-lizard: 10603 Attempt 1: Checking Pool Master Status
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10764 Spawning new instance of iscsi-ha
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10764 check_logger_processes Checking logger processes
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10764 check_logger_processes No processes to clear
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10887 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10887 This iteration is count 12
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10887 Checking if this host is a Pool Master or Slave
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10887 This host's pool status = slave:172.16.215.45
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10922 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10922 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10922 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----     ns:0 nr:362706352 dw:362706352 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10922 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10922 DRBD Resource: iscsi1 in WFConnection state - expected Connected state
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10922 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10922 email: Duplicate message - not sending. Content = DRBD Resource:       iscsi1 in WFConnection state - expected Connected state
Mar 26 09:51:58 xenserver-2646 iscsi-ha: 10922 email: Message barred for 30 minutes
Mar 26 09:52:00 xenserver-2646 iscsid: connect to 192.168.215.44:3260 failed (No route to host) 
Mar 26 09:52:01 xenserver-2646 iscsi-ha-10882-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 172.16.215.45 not responding - replug_pbd exiting
Mar 26 09:52:04 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:52:07 xenserver-2646 iscsid: connect to 192.168.215.44:3260 failed (No route to host) 
Mar 26 09:52:08 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:52:08 xenserver-2646 iscsi-ha: 10879 Spawning new instance of iscsi-ha
Mar 26 09:52:08 xenserver-2646 iscsi-ha: 10879 check_logger_processes Checking logger processes
Mar 26 09:52:08 xenserver-2646 iscsi-ha: 10879 check_logger_processes No processes to clear
Mar 26 09:52:08 xenserver-2646 iscsi-ha: 11035 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:52:08 xenserver-2646 iscsi-ha: 11035 This iteration is count 13
Mar 26 09:52:08 xenserver-2646 iscsi-ha: 11035 Checking if this host is a Pool Master or Slave
Mar 26 09:52:08 xenserver-2646 iscsi-ha: 11035 This host's pool status = slave:172.16.215.45
Mar 26 09:52:08 xenserver-2646 iscsi-ha: 11070 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:52:09 xenserver-2646 iscsi-ha: 11070 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:52:09 xenserver-2646 iscsi-ha: 11070 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----     ns:0 nr:362706352 dw:362706352 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:52:09 xenserver-2646 iscsi-ha: 11070 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:52:09 xenserver-2646 iscsi-ha: 11070 DRBD Resource: iscsi1 in WFConnection state - expected Connected state
Mar 26 09:52:09 xenserver-2646 iscsi-ha: 11070 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:52:09 xenserver-2646 iscsi-ha: 11070 email: Duplicate message - not sending. Content = DRBD Resource:       iscsi1 in WFConnection state - expected Connected state
Mar 26 09:52:09 xenserver-2646 iscsi-ha: 11070 email: Message barred for 30 minutes
Mar 26 09:52:12 xenserver-2646 iscsi-ha-11030-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 172.16.215.45 not responding - replug_pbd exiting
Mar 26 09:52:13 xenserver-2646 iscsid: connect to 192.168.215.44:3260 failed (No route to host) 
Mar 26 09:52:14 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:52:16 xenserver-2646 ha-lizard: 10603 check_xapi: Pool Host 172.16.215.45 failed to respond to ICMP ping
Mar 26 09:52:16 xenserver-2646 ha-lizard: 10603 Attempt 2: Checking Pool Master Status
Mar 26 09:52:17 xenserver-2646 kernel: [53669.962390] igb: eth1 NIC Link is Down
Mar 26 09:52:18 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:52:18 xenserver-2646 iscsi-ha: 11027 Spawning new instance of iscsi-ha
Mar 26 09:52:18 xenserver-2646 iscsi-ha: 11027 check_logger_processes Checking logger processes
Mar 26 09:52:18 xenserver-2646 iscsi-ha: 11027 check_logger_processes No processes to clear
Mar 26 09:52:18 xenserver-2646 iscsi-ha: 11162 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:52:18 xenserver-2646 iscsi-ha: 11162 This iteration is count 14
Mar 26 09:52:18 xenserver-2646 iscsi-ha: 11162 Checking if this host is a Pool Master or Slave
Mar 26 09:52:18 xenserver-2646 iscsi-ha: 11162 This host's pool status = slave:172.16.215.45
Mar 26 09:52:19 xenserver-2646 ha-lizard: 10595 ha-lizard already running: Attempt 1 on PIDS: 10603 10598 10597 10595
Mar 26 09:52:19 xenserver-2646 iscsi-ha: 11203 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:52:19 xenserver-2646 iscsi-ha: 11203 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:52:19 xenserver-2646 iscsi-ha: 11203 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----     ns:0 nr:362706352 dw:362706352 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:52:19 xenserver-2646 iscsi-ha: 11203 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:52:19 xenserver-2646 iscsi-ha: 11203 DRBD Resource: iscsi1 in WFConnection state - expected Connected state
Mar 26 09:52:19 xenserver-2646 iscsi-ha: 11203 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:52:19 xenserver-2646 iscsi-ha: 11203 email: Duplicate message - not sending. Content = DRBD Resource:       iscsi1 in WFConnection state - expected Connected state
Mar 26 09:52:19 xenserver-2646 iscsi-ha: 11203 email: Message barred for 30 minutes
Mar 26 09:52:19 xenserver-2646 iscsid: connect to 192.168.215.44:3260 failed (No route to host) 
Mar 26 09:52:22 xenserver-2646 iscsi-ha-11157-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 172.16.215.45 not responding - replug_pbd exiting
Mar 26 09:52:24 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:52:25 xenserver-2646 iscsid: connect to 192.168.215.44:3260 failed (No route to host) 
Mar 26 09:52:28 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:52:28 xenserver-2646 iscsi-ha: 11154 Spawning new instance of iscsi-ha
Mar 26 09:52:28 xenserver-2646 iscsi-ha: 11154 check_logger_processes Checking logger processes
Mar 26 09:52:29 xenserver-2646 iscsi-ha: 11154 check_logger_processes No processes to clear
Mar 26 09:52:29 xenserver-2646 iscsi-ha: 11281 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:52:29 xenserver-2646 iscsi-ha: 11281 This iteration is count 15
Mar 26 09:52:29 xenserver-2646 iscsi-ha: 11281 Checking if this host is a Pool Master or Slave
Mar 26 09:52:29 xenserver-2646 iscsi-ha: 11281 This host's pool status = slave:172.16.215.45
Mar 26 09:52:29 xenserver-2646 ha-lizard: 10595 ha-lizard already running: Attempt 2 on PIDS: 10603 10598 10597 10595
Mar 26 09:52:29 xenserver-2646 iscsi-ha: 11322 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:52:29 xenserver-2646 iscsi-ha: 11322 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:52:29 xenserver-2646 iscsi-ha: 11322 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----     ns:0 nr:362706352 dw:362706352 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:52:29 xenserver-2646 iscsi-ha: 11322 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:52:29 xenserver-2646 iscsi-ha: 11322 DRBD Resource: iscsi1 in WFConnection state - expected Connected state
Mar 26 09:52:29 xenserver-2646 iscsi-ha: 11322 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:52:29 xenserver-2646 iscsi-ha: 11322 email: Duplicate message - not sending. Content = DRBD Resource:       iscsi1 in WFConnection state - expected Connected state
Mar 26 09:52:29 xenserver-2646 iscsi-ha: 11322 email: Message barred for 30 minutes
Mar 26 09:52:31 xenserver-2646 kernel: [53683.103166] igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Mar 26 09:52:32 xenserver-2646 iscsi-ha-11276-INFO-/etc/iscsi-ha/iscsi-ha.sh: Pool Master 172.16.215.45 not responding - replug_pbd exiting
Mar 26 09:52:32 xenserver-2646 ha-lizard: 10603 check_xapi: Pool Host 172.16.215.45 failed to respond to ICMP ping
Mar 26 09:52:32 xenserver-2646 ha-lizard: 10603 Attempt 3: Checking Pool Master Status
Mar 26 09:52:32 xenserver-2646 iscsid: connect to 192.168.215.44:3260 failed (No route to host) 
Mar 26 09:52:34 xenserver-2646 ha-lizard: 9041 ha-lizard Watchdog: ha-lizard running - OK 
Mar 26 09:52:38 xenserver-2646 iscsi-ha: 8837 iscsi-ha Watchdog: iscsi-ha running - OK 
Mar 26 09:52:38 xenserver-2646 iscsid: connect to 192.168.215.44:3260 failed (No route to host) 
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11273 Spawning new instance of iscsi-ha
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11273 check_logger_processes Checking logger processes
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11273 check_logger_processes No processes to clear
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11404 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:52:39 xenserver-2646 ha-lizard: 10595 ha-lizard already running: Attempt 3 on PIDS: 10603 10598 10597 10595
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11404 This iteration is count 16
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11404 Checking if this host is a Pool Master or Slave
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11404 This host's pool status = slave:172.16.215.45
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11443 local_ip_list: Local IP list returned 127.0.0.1 172.16.215.46 192.168.215.46
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11443 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11443 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@localhost.localdomain, 2013-12-19 20:12:03   1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----     ns:0 nr:362706352 dw:362706352 dr:0 al:0 bm:22165 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11443 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11443 DRBD Resource: iscsi1 in WFConnection state - expected Connected state
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11443 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11443 email: Duplicate message - not sending. Content = DRBD Resource:       iscsi1 in WFConnection state - expected Connected state
Mar 26 09:52:39 xenserver-2646 iscsi-ha: 11443 email: Message barred for 30 minutes