Dec 16 09:27:23 xen-02 iscsi-ha: 4379 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:27:23 xen-02 iscsi-ha: 4379 DRBD Resource: iscsi1 in Connected state
Dec 16 09:27:23 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:27:23 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:27:24 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:27:26 xen-02 ha-lizard: 3395 Spawning new instance of ha-lizard
Dec 16 09:27:26 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:27:26 xen-02 ha-lizard:  This iteration is count 4426
Dec 16 09:27:26 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:27:26 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:27:26 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:27:26 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:27:26 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:27:26 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:27:26 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:27:26 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:27:27 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:27:27 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:27:27 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:27:27 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:27:28 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:27:28 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:27:28 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:27:28 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:27:28 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:27:28 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:27:28 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:27:28 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:27:28 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:27:28 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:27:28 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:27:28 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:27:28 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:27:28 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:27:29 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:27:29 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:27:29 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:27:29 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:27:29 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:27:29 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:27:29 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:27:29 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:27:30 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:27:30 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:27:31 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:27:32 xen-02 iscsi-ha: 4336 Spawning new instance of iscsi-ha
Dec 16 09:27:32 xen-02 iscsi-ha: 4336 check_logger_processes Checking logger processes
Dec 16 09:27:32 xen-02 iscsi-ha: 4336 check_logger_processes No processes to clear
Dec 16 09:27:32 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:27:32 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:27:33 xen-02 iscsi-ha:  This iteration is count 6602
Dec 16 09:27:33 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:27:33 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:27:33 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:27:33 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:27:33 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:27:33 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:27:33 xen-02 iscsi-ha: 6303 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:27:33 xen-02 iscsi-ha: 6303 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:27:33 xen-02 iscsi-ha: 6303 service_execute: systemctl mode being used
Dec 16 09:27:33 xen-02 iscsi-ha: 6303 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:27:33 xen-02 iscsi-ha: 6303 service_execute: Returning exit status [ 3 ]
Dec 16 09:27:33 xen-02 iscsi-ha: 6303 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:27:33 xen-02 iscsi-ha: 6303 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485356 dw:1087485356 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:27:33 xen-02 iscsi-ha: 6303 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:27:33 xen-02 iscsi-ha: 6303 DRBD Resource: iscsi1 in Connected state
Dec 16 09:27:33 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:27:33 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:27:34 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:27:36 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:27:41 xen-02 ha-lizard: 5077 Spawning new instance of ha-lizard
Dec 16 09:27:41 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:27:41 xen-02 ha-lizard:  This iteration is count 4427
Dec 16 09:27:41 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:27:41 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:27:41 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:27:41 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:27:41 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:27:41 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:27:41 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:27:41 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:27:42 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:27:43 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:27:43 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:27:43 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:27:43 xen-02 iscsi-ha: 6242 Spawning new instance of iscsi-ha
Dec 16 09:27:43 xen-02 iscsi-ha: 6242 check_logger_processes Checking logger processes
Dec 16 09:27:43 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:27:43 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:27:43 xen-02 iscsi-ha: 6242 check_logger_processes No processes to clear
Dec 16 09:27:43 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:27:43 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:27:43 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:27:43 xen-02 iscsi-ha:  This iteration is count 6603
Dec 16 09:27:43 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:27:43 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:27:43 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:27:43 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:27:43 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:27:43 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:27:43 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:27:43 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:27:43 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:27:43 xen-02 iscsi-ha: 6572 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:27:43 xen-02 iscsi-ha: 6572 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:27:43 xen-02 iscsi-ha: 6572 service_execute: systemctl mode being used
Dec 16 09:27:43 xen-02 iscsi-ha: 6572 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:27:43 xen-02 iscsi-ha: 6572 service_execute: Returning exit status [ 3 ]
Dec 16 09:27:43 xen-02 iscsi-ha: 6572 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:27:43 xen-02 iscsi-ha: 6572 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485356 dw:1087485356 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:27:43 xen-02 iscsi-ha: 6572 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:27:43 xen-02 iscsi-ha: 6572 DRBD Resource: iscsi1 in Connected state
Dec 16 09:27:43 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:27:43 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:27:43 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:27:43 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:27:43 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:27:43 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:27:43 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:27:43 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:27:43 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:27:43 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:27:44 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:27:44 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:27:44 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:27:44 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:27:44 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:27:44 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:27:44 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:27:44 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:27:45 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:27:45 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:27:45 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:27:46 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:27:51 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:27:53 xen-02 iscsi-ha: 6523 Spawning new instance of iscsi-ha
Dec 16 09:27:53 xen-02 iscsi-ha: 6523 check_logger_processes Checking logger processes
Dec 16 09:27:53 xen-02 iscsi-ha: 6523 check_logger_processes No processes to clear
Dec 16 09:27:53 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:27:53 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:27:53 xen-02 iscsi-ha:  This iteration is count 6604
Dec 16 09:27:53 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:27:53 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:27:53 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:27:53 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:27:53 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:27:53 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:27:53 xen-02 iscsi-ha: 7259 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:27:53 xen-02 iscsi-ha: 7259 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:27:53 xen-02 iscsi-ha: 7259 service_execute: systemctl mode being used
Dec 16 09:27:53 xen-02 iscsi-ha: 7259 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:27:53 xen-02 iscsi-ha: 7259 service_execute: Returning exit status [ 3 ]
Dec 16 09:27:53 xen-02 iscsi-ha: 7259 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:27:53 xen-02 iscsi-ha: 7259 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485364 dw:1087485364 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:27:53 xen-02 iscsi-ha: 7259 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:27:53 xen-02 iscsi-ha: 7259 DRBD Resource: iscsi1 in Connected state
Dec 16 09:27:53 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:27:53 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:27:54 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:27:56 xen-02 ha-lizard: 6412 Spawning new instance of ha-lizard
Dec 16 09:27:56 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:27:56 xen-02 ha-lizard:  This iteration is count 4428
Dec 16 09:27:56 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:27:56 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:27:56 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:27:57 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:27:57 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:27:57 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:27:57 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:27:57 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:27:58 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:27:58 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:27:58 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:27:58 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:27:58 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:27:58 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:27:58 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:27:58 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:27:58 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:27:58 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:27:58 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:27:58 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:27:58 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:27:58 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:27:58 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:27:58 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:27:58 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:27:58 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:27:59 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:27:59 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:27:59 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:27:59 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:27:59 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:28:00 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:28:00 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:28:00 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:28:00 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:28:00 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:28:01 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:28:03 xen-02 iscsi-ha: 7224 Spawning new instance of iscsi-ha
Dec 16 09:28:03 xen-02 iscsi-ha: 7224 check_logger_processes Checking logger processes
Dec 16 09:28:03 xen-02 iscsi-ha: 7224 check_logger_processes No processes to clear
Dec 16 09:28:03 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:28:03 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:28:03 xen-02 iscsi-ha:  This iteration is count 6605
Dec 16 09:28:03 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:28:03 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:28:03 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:28:03 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:28:03 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:28:03 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:28:03 xen-02 iscsi-ha: 8862 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:28:03 xen-02 iscsi-ha: 8862 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:28:03 xen-02 iscsi-ha: 8862 service_execute: systemctl mode being used
Dec 16 09:28:03 xen-02 iscsi-ha: 8862 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:28:03 xen-02 iscsi-ha: 8862 service_execute: Returning exit status [ 3 ]
Dec 16 09:28:03 xen-02 iscsi-ha: 8862 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:28:03 xen-02 iscsi-ha: 8862 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485364 dw:1087485364 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:28:03 xen-02 iscsi-ha: 8862 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:28:03 xen-02 iscsi-ha: 8862 DRBD Resource: iscsi1 in Connected state
Dec 16 09:28:03 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:28:03 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:28:04 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:28:06 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:28:11 xen-02 ha-lizard: 7349 Spawning new instance of ha-lizard
Dec 16 09:28:11 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:28:11 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:28:11 xen-02 ha-lizard:  This iteration is count 4429
Dec 16 09:28:11 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:28:11 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:28:12 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:28:12 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:28:12 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:28:12 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:28:12 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:28:13 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:28:13 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:28:13 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:28:13 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:28:13 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:28:13 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:28:13 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:28:13 xen-02 iscsi-ha: 8803 Spawning new instance of iscsi-ha
Dec 16 09:28:13 xen-02 iscsi-ha: 8803 check_logger_processes Checking logger processes
Dec 16 09:28:13 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:28:13 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:28:13 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:28:13 xen-02 iscsi-ha: 8803 check_logger_processes No processes to clear
Dec 16 09:28:13 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:28:13 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:28:13 xen-02 iscsi-ha:  This iteration is count 6606
Dec 16 09:28:13 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:28:13 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:28:13 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:28:13 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:28:13 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:28:13 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:28:13 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:28:13 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:28:13 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:28:13 xen-02 iscsi-ha: 9202 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:28:13 xen-02 iscsi-ha: 9202 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:28:13 xen-02 iscsi-ha: 9202 service_execute: systemctl mode being used
Dec 16 09:28:13 xen-02 iscsi-ha: 9202 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:28:13 xen-02 iscsi-ha: 9202 service_execute: Returning exit status [ 3 ]
Dec 16 09:28:13 xen-02 iscsi-ha: 9202 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:28:13 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:28:13 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:28:13 xen-02 iscsi-ha: 9202 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485380 dw:1087485380 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:28:13 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:28:13 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:28:13 xen-02 iscsi-ha: 9202 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:28:13 xen-02 iscsi-ha: 9202 DRBD Resource: iscsi1 in Connected state
Dec 16 09:28:13 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:28:13 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:28:14 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:28:14 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:28:14 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:28:14 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:28:14 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:28:14 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:28:14 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:28:15 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:28:15 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:28:15 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:28:15 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:28:15 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:28:16 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:28:21 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:28:23 xen-02 iscsi-ha: 9137 Spawning new instance of iscsi-ha
Dec 16 09:28:23 xen-02 iscsi-ha: 9137 check_logger_processes Checking logger processes
Dec 16 09:28:23 xen-02 iscsi-ha: 9137 check_logger_processes No processes to clear
Dec 16 09:28:23 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:28:23 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:28:23 xen-02 iscsi-ha:  This iteration is count 6607
Dec 16 09:28:23 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:28:23 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:28:23 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:28:23 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:28:23 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:28:23 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:28:23 xen-02 iscsi-ha: 9668 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:28:23 xen-02 iscsi-ha: 9668 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:28:23 xen-02 iscsi-ha: 9668 service_execute: systemctl mode being used
Dec 16 09:28:23 xen-02 iscsi-ha: 9668 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:28:23 xen-02 iscsi-ha: 9668 service_execute: Returning exit status [ 3 ]
Dec 16 09:28:24 xen-02 iscsi-ha: 9668 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:28:24 xen-02 iscsi-ha: 9668 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485380 dw:1087485380 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:28:24 xen-02 iscsi-ha: 9668 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:28:24 xen-02 iscsi-ha: 9668 DRBD Resource: iscsi1 in Connected state
Dec 16 09:28:24 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:28:24 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:28:24 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:28:26 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:28:26 xen-02 ha-lizard: 9012 Spawning new instance of ha-lizard
Dec 16 09:28:27 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:28:27 xen-02 ha-lizard:  This iteration is count 4430
Dec 16 09:28:27 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:28:27 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:28:27 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:28:27 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:28:27 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:28:27 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:28:27 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:28:28 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:28:28 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:28:28 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:28:28 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:28:28 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:28:28 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:28:28 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:28:28 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:28:28 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:28:28 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:28:28 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:28:28 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:28:28 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:28:29 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:28:29 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:28:29 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:28:29 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:28:29 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:28:29 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:28:29 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:28:29 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:28:29 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:28:29 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:28:30 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:28:30 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:28:30 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:28:30 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:28:30 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:28:31 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:28:33 xen-02 iscsi-ha: 9633 Spawning new instance of iscsi-ha
Dec 16 09:28:33 xen-02 iscsi-ha: 9633 check_logger_processes Checking logger processes
Dec 16 09:28:33 xen-02 iscsi-ha: 9633 check_logger_processes No processes to clear
Dec 16 09:28:33 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:28:33 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:28:33 xen-02 iscsi-ha:  This iteration is count 6608
Dec 16 09:28:33 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:28:33 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:28:33 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:28:33 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:28:34 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:28:34 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:28:34 xen-02 iscsi-ha: 10464 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:28:34 xen-02 iscsi-ha: 10464 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:28:34 xen-02 iscsi-ha: 10464 service_execute: systemctl mode being used
Dec 16 09:28:34 xen-02 iscsi-ha: 10464 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:28:34 xen-02 iscsi-ha: 10464 service_execute: Returning exit status [ 3 ]
Dec 16 09:28:34 xen-02 iscsi-ha: 10464 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:28:34 xen-02 iscsi-ha: 10464 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485380 dw:1087485380 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:28:34 xen-02 iscsi-ha: 10464 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:28:34 xen-02 iscsi-ha: 10464 DRBD Resource: iscsi1 in Connected state
Dec 16 09:28:34 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:28:34 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:28:34 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:28:36 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:28:41 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:28:42 xen-02 ha-lizard: 9939 Spawning new instance of ha-lizard
Dec 16 09:28:42 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:28:42 xen-02 ha-lizard:  This iteration is count 4431
Dec 16 09:28:42 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:28:42 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:28:42 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:28:42 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:28:42 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:28:42 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:28:42 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:28:43 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:28:43 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:28:43 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:28:43 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:28:43 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:28:43 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:28:43 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:28:43 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:28:43 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:28:43 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:28:43 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:28:43 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:28:43 xen-02 iscsi-ha: 10413 Spawning new instance of iscsi-ha
Dec 16 09:28:43 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:28:44 xen-02 iscsi-ha: 10413 check_logger_processes Checking logger processes
Dec 16 09:28:44 xen-02 iscsi-ha: 10413 check_logger_processes No processes to clear
Dec 16 09:28:44 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:28:44 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:28:44 xen-02 iscsi-ha:  This iteration is count 6609
Dec 16 09:28:44 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:28:44 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:28:44 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:28:44 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:28:44 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:28:44 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:28:44 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:28:44 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:28:44 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:28:44 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:28:44 xen-02 iscsi-ha: 11798 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:28:44 xen-02 iscsi-ha: 11798 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:28:44 xen-02 iscsi-ha: 11798 service_execute: systemctl mode being used
Dec 16 09:28:44 xen-02 iscsi-ha: 11798 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:28:44 xen-02 iscsi-ha: 11798 service_execute: Returning exit status [ 3 ]
Dec 16 09:28:44 xen-02 iscsi-ha: 11798 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:28:44 xen-02 iscsi-ha: 11798 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485388 dw:1087485388 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:28:44 xen-02 iscsi-ha: 11798 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:28:44 xen-02 iscsi-ha: 11798 DRBD Resource: iscsi1 in Connected state
Dec 16 09:28:44 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:28:44 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:28:44 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:28:44 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:28:44 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:28:44 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:28:45 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:28:45 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:28:45 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:28:45 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:28:45 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:28:45 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:28:45 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:28:45 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:28:46 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:28:52 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:28:54 xen-02 iscsi-ha: 11736 Spawning new instance of iscsi-ha
Dec 16 09:28:54 xen-02 iscsi-ha: 11736 check_logger_processes Checking logger processes
Dec 16 09:28:54 xen-02 iscsi-ha: 11736 check_logger_processes No processes to clear
Dec 16 09:28:54 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:28:54 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:28:54 xen-02 iscsi-ha:  This iteration is count 6610
Dec 16 09:28:54 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:28:54 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:28:54 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:28:54 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:28:54 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:28:54 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:28:54 xen-02 iscsi-ha: 12236 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:28:54 xen-02 iscsi-ha: 12236 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:28:54 xen-02 iscsi-ha: 12236 service_execute: systemctl mode being used
Dec 16 09:28:54 xen-02 iscsi-ha: 12236 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:28:54 xen-02 iscsi-ha: 12236 service_execute: Returning exit status [ 3 ]
Dec 16 09:28:54 xen-02 iscsi-ha: 12236 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:28:54 xen-02 iscsi-ha: 12236 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485388 dw:1087485388 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:28:54 xen-02 iscsi-ha: 12236 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:28:54 xen-02 iscsi-ha: 12236 DRBD Resource: iscsi1 in Connected state
Dec 16 09:28:54 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:28:54 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:28:54 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:28:57 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:28:57 xen-02 ha-lizard: 11577 Spawning new instance of ha-lizard
Dec 16 09:28:57 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:28:57 xen-02 ha-lizard:  This iteration is count 4432
Dec 16 09:28:57 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:28:57 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:28:57 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:28:57 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:28:57 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:28:57 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:28:57 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:28:58 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:28:58 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:28:58 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:28:58 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:28:58 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:28:58 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:28:58 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:28:58 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:28:58 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:28:58 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:28:59 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:28:59 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:28:59 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:28:59 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:28:59 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:28:59 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:28:59 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:28:59 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:28:59 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:29:00 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:29:00 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:29:00 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:29:00 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:29:00 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:29:00 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:29:00 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:29:00 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:29:00 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:29:02 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:29:04 xen-02 iscsi-ha: 12194 Spawning new instance of iscsi-ha
Dec 16 09:29:04 xen-02 iscsi-ha: 12194 check_logger_processes Checking logger processes
Dec 16 09:29:04 xen-02 iscsi-ha: 12194 check_logger_processes No processes to clear
Dec 16 09:29:04 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:29:04 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:29:04 xen-02 iscsi-ha:  This iteration is count 6611
Dec 16 09:29:04 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:29:04 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:29:04 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:29:04 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:29:04 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:29:04 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:29:04 xen-02 iscsi-ha: 13015 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:29:04 xen-02 iscsi-ha: 13015 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:29:04 xen-02 iscsi-ha: 13015 service_execute: systemctl mode being used
Dec 16 09:29:04 xen-02 iscsi-ha: 13015 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:29:04 xen-02 iscsi-ha: 13015 service_execute: Returning exit status [ 3 ]
Dec 16 09:29:04 xen-02 iscsi-ha: 13015 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:29:04 xen-02 iscsi-ha: 13015 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485388 dw:1087485388 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:29:04 xen-02 iscsi-ha: 13015 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:29:04 xen-02 iscsi-ha: 13015 DRBD Resource: iscsi1 in Connected state
Dec 16 09:29:04 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:29:04 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:29:04 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:29:07 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:29:12 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:29:12 xen-02 ha-lizard: 12502 Spawning new instance of ha-lizard
Dec 16 09:29:12 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:29:12 xen-02 ha-lizard:  This iteration is count 4433
Dec 16 09:29:12 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:29:12 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:29:12 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:29:12 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:29:12 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:29:12 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:29:12 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:29:13 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:29:13 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:29:13 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:29:13 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:29:13 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:29:13 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:29:13 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:29:14 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:29:14 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:29:14 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:29:14 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:29:14 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:29:14 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:29:14 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:29:14 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:29:14 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:29:14 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:29:14 xen-02 iscsi-ha: 12973 Spawning new instance of iscsi-ha
Dec 16 09:29:14 xen-02 iscsi-ha: 12973 check_logger_processes Checking logger processes
Dec 16 09:29:14 xen-02 iscsi-ha: 12973 check_logger_processes No processes to clear
Dec 16 09:29:14 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:29:14 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:29:14 xen-02 iscsi-ha:  This iteration is count 6612
Dec 16 09:29:14 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:29:14 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:29:14 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:29:14 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:29:14 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:29:14 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:29:14 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:29:14 xen-02 iscsi-ha: 14367 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:29:14 xen-02 iscsi-ha: 14367 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:29:14 xen-02 iscsi-ha: 14367 service_execute: systemctl mode being used
Dec 16 09:29:14 xen-02 iscsi-ha: 14367 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:29:14 xen-02 iscsi-ha: 14367 service_execute: Returning exit status [ 3 ]
Dec 16 09:29:14 xen-02 iscsi-ha: 14367 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:29:14 xen-02 iscsi-ha: 14367 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485404 dw:1087485404 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:29:14 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:29:14 xen-02 iscsi-ha: 14367 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:29:14 xen-02 iscsi-ha: 14367 DRBD Resource: iscsi1 in Connected state
Dec 16 09:29:14 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:29:14 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:29:15 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:29:15 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:29:15 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:29:15 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:29:15 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:29:15 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:29:15 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:29:15 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:29:15 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:29:15 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:29:17 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:29:22 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:29:24 xen-02 iscsi-ha: 14318 Spawning new instance of iscsi-ha
Dec 16 09:29:24 xen-02 iscsi-ha: 14318 check_logger_processes Checking logger processes
Dec 16 09:29:24 xen-02 iscsi-ha: 14318 check_logger_processes No processes to clear
Dec 16 09:29:24 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:29:24 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:29:24 xen-02 iscsi-ha:  This iteration is count 6613
Dec 16 09:29:24 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:29:24 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:29:24 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:29:24 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:29:24 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:29:24 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:29:24 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:29:24 xen-02 iscsi-ha: 14781 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:29:24 xen-02 iscsi-ha: 14781 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:29:24 xen-02 iscsi-ha: 14781 service_execute: systemctl mode being used
Dec 16 09:29:24 xen-02 iscsi-ha: 14781 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:29:24 xen-02 iscsi-ha: 14781 service_execute: Returning exit status [ 3 ]
Dec 16 09:29:24 xen-02 iscsi-ha: 14781 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:29:24 xen-02 iscsi-ha: 14781 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485404 dw:1087485404 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:29:24 xen-02 iscsi-ha: 14781 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:29:24 xen-02 iscsi-ha: 14781 DRBD Resource: iscsi1 in Connected state
Dec 16 09:29:25 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:29:25 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:29:27 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:29:27 xen-02 ha-lizard: 13850 Spawning new instance of ha-lizard
Dec 16 09:29:27 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:29:27 xen-02 ha-lizard:  This iteration is count 4434
Dec 16 09:29:27 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:29:27 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:29:27 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:29:27 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:29:27 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:29:27 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:29:27 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:29:28 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:29:28 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:29:28 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:29:28 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:29:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:29:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:29:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:29:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:29:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:29:29 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:29:29 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:29:29 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:29:29 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:29:29 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:29:29 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:29:29 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:29:29 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:29:29 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:29:30 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:29:30 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:29:30 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:29:30 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:29:30 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:29:30 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:29:30 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:29:30 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:29:31 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:29:31 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:29:32 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:29:34 xen-02 iscsi-ha: 14744 Spawning new instance of iscsi-ha
Dec 16 09:29:34 xen-02 iscsi-ha: 14744 check_logger_processes Checking logger processes
Dec 16 09:29:34 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:29:34 xen-02 iscsi-ha: 14744 check_logger_processes No processes to clear
Dec 16 09:29:34 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:29:34 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:29:34 xen-02 iscsi-ha:  This iteration is count 6614
Dec 16 09:29:34 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:29:34 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:29:34 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:29:34 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:29:34 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:29:34 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:29:35 xen-02 iscsi-ha: 15561 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:29:35 xen-02 iscsi-ha: 15561 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:29:35 xen-02 iscsi-ha: 15561 service_execute: systemctl mode being used
Dec 16 09:29:35 xen-02 iscsi-ha: 15561 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:29:35 xen-02 iscsi-ha: 15561 service_execute: Returning exit status [ 3 ]
Dec 16 09:29:35 xen-02 iscsi-ha: 15561 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:29:35 xen-02 iscsi-ha: 15561 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485404 dw:1087485404 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:29:35 xen-02 iscsi-ha: 15561 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:29:35 xen-02 iscsi-ha: 15561 DRBD Resource: iscsi1 in Connected state
Dec 16 09:29:35 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:29:35 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:29:37 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:29:42 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:29:42 xen-02 ha-lizard: 15028 Spawning new instance of ha-lizard
Dec 16 09:29:42 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:29:42 xen-02 ha-lizard:  This iteration is count 4435
Dec 16 09:29:42 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:29:42 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:29:43 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:29:43 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:29:43 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:29:43 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:29:43 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:29:44 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:29:44 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:29:44 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:29:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:29:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:29:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:29:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:29:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:29:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:29:44 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:29:44 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:29:44 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:29:44 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:29:44 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:29:44 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:29:44 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:29:44 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:29:44 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:29:44 xen-02 iscsi-ha: 15526 Spawning new instance of iscsi-ha
Dec 16 09:29:44 xen-02 iscsi-ha: 15526 check_logger_processes Checking logger processes
Dec 16 09:29:44 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:29:44 xen-02 iscsi-ha: 15526 check_logger_processes No processes to clear
Dec 16 09:29:45 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:29:45 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:29:45 xen-02 iscsi-ha:  This iteration is count 6615
Dec 16 09:29:45 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:29:45 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:29:45 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:29:45 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:29:45 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:29:45 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:29:45 xen-02 iscsi-ha: 16180 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:29:45 xen-02 iscsi-ha: 16180 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:29:45 xen-02 iscsi-ha: 16180 service_execute: systemctl mode being used
Dec 16 09:29:45 xen-02 iscsi-ha: 16180 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:29:45 xen-02 iscsi-ha: 16180 service_execute: Returning exit status [ 3 ]
Dec 16 09:29:45 xen-02 iscsi-ha: 16180 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:29:45 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:29:45 xen-02 iscsi-ha: 16180 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485412 dw:1087485412 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:29:45 xen-02 iscsi-ha: 16180 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:29:45 xen-02 iscsi-ha: 16180 DRBD Resource: iscsi1 in Connected state
Dec 16 09:29:45 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:29:45 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:29:45 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:29:45 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:29:45 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:29:45 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:29:45 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:29:46 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:29:46 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:29:46 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:29:46 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:29:47 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:29:52 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:29:54 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:29:55 xen-02 iscsi-ha: 16078 Spawning new instance of iscsi-ha
Dec 16 09:29:55 xen-02 iscsi-ha: 16078 check_logger_processes Checking logger processes
Dec 16 09:29:55 xen-02 iscsi-ha: 16078 check_logger_processes No processes to clear
Dec 16 09:29:55 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:29:55 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:29:55 xen-02 iscsi-ha:  This iteration is count 6616
Dec 16 09:29:55 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:29:55 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:29:55 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:29:55 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:29:55 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:29:55 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:29:55 xen-02 iscsi-ha: 17350 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:29:55 xen-02 iscsi-ha: 17350 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:29:55 xen-02 iscsi-ha: 17350 service_execute: systemctl mode being used
Dec 16 09:29:55 xen-02 iscsi-ha: 17350 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:29:55 xen-02 iscsi-ha: 17350 service_execute: Returning exit status [ 3 ]
Dec 16 09:29:55 xen-02 iscsi-ha: 17350 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:29:55 xen-02 iscsi-ha: 17350 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485412 dw:1087485412 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:29:55 xen-02 iscsi-ha: 17350 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:29:55 xen-02 iscsi-ha: 17350 DRBD Resource: iscsi1 in Connected state
Dec 16 09:29:55 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:29:55 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:29:57 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:29:57 xen-02 ha-lizard: 15672 Spawning new instance of ha-lizard
Dec 16 09:29:57 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:29:57 xen-02 ha-lizard:  This iteration is count 4436
Dec 16 09:29:57 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:29:57 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:29:58 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:29:58 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:29:58 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:29:58 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:29:58 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:29:59 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:29:59 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:29:59 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:29:59 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:29:59 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:29:59 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:29:59 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:29:59 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:29:59 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:29:59 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:29:59 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:29:59 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:29:59 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:29:59 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:29:59 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:29:59 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:29:59 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:30:00 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:30:00 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:30:00 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:30:00 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:30:00 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:30:00 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:30:01 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:30:01 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:30:01 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:30:01 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:30:01 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:30:02 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:30:04 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:30:05 xen-02 iscsi-ha: 17315 Spawning new instance of iscsi-ha
Dec 16 09:30:05 xen-02 iscsi-ha: 17315 check_logger_processes Checking logger processes
Dec 16 09:30:05 xen-02 iscsi-ha: 17315 check_logger_processes No processes to clear
Dec 16 09:30:05 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:30:05 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:30:05 xen-02 iscsi-ha:  This iteration is count 6617
Dec 16 09:30:05 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:30:05 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:30:05 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:30:05 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:30:05 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:30:05 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:30:05 xen-02 iscsi-ha: 18146 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:30:05 xen-02 iscsi-ha: 18146 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:30:05 xen-02 iscsi-ha: 18146 service_execute: systemctl mode being used
Dec 16 09:30:05 xen-02 iscsi-ha: 18146 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:30:05 xen-02 iscsi-ha: 18146 service_execute: Returning exit status [ 3 ]
Dec 16 09:30:05 xen-02 iscsi-ha: 18146 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:30:05 xen-02 iscsi-ha: 18146 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485412 dw:1087485412 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:30:05 xen-02 iscsi-ha: 18146 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:30:05 xen-02 iscsi-ha: 18146 DRBD Resource: iscsi1 in Connected state
Dec 16 09:30:05 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:30:05 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:30:07 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:30:12 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:30:12 xen-02 ha-lizard: 17441 Spawning new instance of ha-lizard
Dec 16 09:30:13 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:30:13 xen-02 ha-lizard:  This iteration is count 4437
Dec 16 09:30:13 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:30:13 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:30:13 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:30:13 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:30:13 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:30:13 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:30:13 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:30:14 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:30:14 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:30:14 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:30:14 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:30:14 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:30:14 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:30:14 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:30:14 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:30:14 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:30:14 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:30:14 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:30:14 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:30:14 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:30:14 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:30:14 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:30:14 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:30:14 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:30:14 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:30:15 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:30:15 xen-02 iscsi-ha: 18111 Spawning new instance of iscsi-ha
Dec 16 09:30:15 xen-02 iscsi-ha: 18111 check_logger_processes Checking logger processes
Dec 16 09:30:15 xen-02 iscsi-ha: 18111 check_logger_processes No processes to clear
Dec 16 09:30:15 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:30:15 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:30:15 xen-02 iscsi-ha:  This iteration is count 6618
Dec 16 09:30:15 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:30:15 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:30:15 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:30:15 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:30:15 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:30:15 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:30:15 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:30:15 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:30:15 xen-02 iscsi-ha: 18613 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:30:15 xen-02 iscsi-ha: 18613 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:30:15 xen-02 iscsi-ha: 18613 service_execute: systemctl mode being used
Dec 16 09:30:15 xen-02 iscsi-ha: 18613 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:30:15 xen-02 iscsi-ha: 18613 service_execute: Returning exit status [ 3 ]
Dec 16 09:30:15 xen-02 iscsi-ha: 18613 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:30:15 xen-02 iscsi-ha: 18613 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485428 dw:1087485428 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:30:15 xen-02 iscsi-ha: 18613 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:30:15 xen-02 iscsi-ha: 18613 DRBD Resource: iscsi1 in Connected state
Dec 16 09:30:15 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:30:15 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:30:15 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:30:15 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:30:15 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:30:16 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:30:16 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:30:16 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:30:16 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:30:16 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:30:17 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:30:22 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:30:24 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:30:25 xen-02 iscsi-ha: 18556 Spawning new instance of iscsi-ha
Dec 16 09:30:25 xen-02 iscsi-ha: 18556 check_logger_processes Checking logger processes
Dec 16 09:30:25 xen-02 iscsi-ha: 18556 check_logger_processes No processes to clear
Dec 16 09:30:25 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:30:25 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:30:25 xen-02 iscsi-ha:  This iteration is count 6619
Dec 16 09:30:25 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:30:25 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:30:25 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:30:25 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:30:25 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:30:25 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:30:25 xen-02 iscsi-ha: 20040 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:30:25 xen-02 iscsi-ha: 20040 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:30:25 xen-02 iscsi-ha: 20040 service_execute: systemctl mode being used
Dec 16 09:30:25 xen-02 iscsi-ha: 20040 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:30:25 xen-02 iscsi-ha: 20040 service_execute: Returning exit status [ 3 ]
Dec 16 09:30:25 xen-02 iscsi-ha: 20040 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:30:25 xen-02 iscsi-ha: 20040 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485428 dw:1087485428 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:30:25 xen-02 iscsi-ha: 20040 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:30:25 xen-02 iscsi-ha: 20040 DRBD Resource: iscsi1 in Connected state
Dec 16 09:30:25 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:30:25 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:30:27 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:30:28 xen-02 ha-lizard: 18329 Spawning new instance of ha-lizard
Dec 16 09:30:28 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:30:28 xen-02 ha-lizard:  This iteration is count 4438
Dec 16 09:30:28 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:30:28 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:30:28 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:30:28 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:30:28 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:30:28 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:30:28 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:30:29 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:30:29 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:30:29 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:30:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:30:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:30:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:30:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:30:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:30:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:30:29 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:30:29 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:30:29 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:30:29 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:30:30 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:30:30 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:30:30 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:30:30 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:30:30 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:30:30 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:30:30 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:30:31 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:30:31 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:30:31 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:30:31 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:30:31 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:30:31 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:30:31 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:30:31 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:30:32 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:30:34 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:30:35 xen-02 iscsi-ha: 19993 Spawning new instance of iscsi-ha
Dec 16 09:30:35 xen-02 iscsi-ha: 19993 check_logger_processes Checking logger processes
Dec 16 09:30:35 xen-02 iscsi-ha: 19993 check_logger_processes No processes to clear
Dec 16 09:30:35 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:30:35 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:30:35 xen-02 iscsi-ha:  This iteration is count 6620
Dec 16 09:30:35 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:30:35 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:30:35 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:30:35 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:30:35 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:30:35 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:30:35 xen-02 iscsi-ha: 20827 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:30:35 xen-02 iscsi-ha: 20827 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:30:35 xen-02 iscsi-ha: 20827 service_execute: systemctl mode being used
Dec 16 09:30:35 xen-02 iscsi-ha: 20827 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:30:36 xen-02 iscsi-ha: 20827 service_execute: Returning exit status [ 3 ]
Dec 16 09:30:36 xen-02 iscsi-ha: 20827 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:30:36 xen-02 iscsi-ha: 20827 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485428 dw:1087485428 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:30:36 xen-02 iscsi-ha: 20827 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:30:36 xen-02 iscsi-ha: 20827 DRBD Resource: iscsi1 in Connected state
Dec 16 09:30:36 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:30:36 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:30:37 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:30:42 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:30:43 xen-02 ha-lizard: 20114 Spawning new instance of ha-lizard
Dec 16 09:30:43 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:30:43 xen-02 ha-lizard:  This iteration is count 4439
Dec 16 09:30:43 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:30:43 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:30:43 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:30:43 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:30:43 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:30:43 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:30:43 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:30:44 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:30:44 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:30:44 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:30:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:30:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:30:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:30:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:30:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:30:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:30:44 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:30:44 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:30:45 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:30:45 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:30:45 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:30:45 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:30:45 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:30:45 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:30:45 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:30:45 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:30:45 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:30:45 xen-02 iscsi-ha: 20791 Spawning new instance of iscsi-ha
Dec 16 09:30:45 xen-02 iscsi-ha: 20791 check_logger_processes Checking logger processes
Dec 16 09:30:45 xen-02 iscsi-ha: 20791 check_logger_processes No processes to clear
Dec 16 09:30:45 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:30:45 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:30:45 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:30:45 xen-02 iscsi-ha:  This iteration is count 6621
Dec 16 09:30:45 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:30:45 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:30:45 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:30:45 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:30:46 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:30:46 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:30:46 xen-02 iscsi-ha: 21251 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:30:46 xen-02 iscsi-ha: 21251 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:30:46 xen-02 iscsi-ha: 21251 service_execute: systemctl mode being used
Dec 16 09:30:46 xen-02 iscsi-ha: 21251 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:30:46 xen-02 iscsi-ha: 21251 service_execute: Returning exit status [ 3 ]
Dec 16 09:30:46 xen-02 iscsi-ha: 21251 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:30:46 xen-02 iscsi-ha: 21251 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485436 dw:1087485436 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:30:46 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:30:46 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:30:46 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:30:46 xen-02 iscsi-ha: 21251 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:30:46 xen-02 iscsi-ha: 21251 DRBD Resource: iscsi1 in Connected state
Dec 16 09:30:46 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:30:46 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:30:46 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:30:46 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:30:46 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:30:46 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:30:46 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:30:47 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:30:52 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:30:54 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:30:56 xen-02 iscsi-ha: 21186 Spawning new instance of iscsi-ha
Dec 16 09:30:56 xen-02 iscsi-ha: 21186 check_logger_processes Checking logger processes
Dec 16 09:30:56 xen-02 iscsi-ha: 21186 check_logger_processes No processes to clear
Dec 16 09:30:56 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:30:56 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:30:56 xen-02 iscsi-ha:  This iteration is count 6622
Dec 16 09:30:56 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:30:56 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:30:56 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:30:56 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:30:56 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:30:56 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:30:56 xen-02 iscsi-ha: 22054 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:30:56 xen-02 iscsi-ha: 22054 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:30:56 xen-02 iscsi-ha: 22054 service_execute: systemctl mode being used
Dec 16 09:30:56 xen-02 iscsi-ha: 22054 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:30:56 xen-02 iscsi-ha: 22054 service_execute: Returning exit status [ 3 ]
Dec 16 09:30:56 xen-02 iscsi-ha: 22054 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:30:56 xen-02 iscsi-ha: 22054 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485436 dw:1087485436 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:30:56 xen-02 iscsi-ha: 22054 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:30:56 xen-02 iscsi-ha: 22054 DRBD Resource: iscsi1 in Connected state
Dec 16 09:30:56 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:30:56 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:30:57 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:30:58 xen-02 ha-lizard: 20918 Spawning new instance of ha-lizard
Dec 16 09:30:58 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:30:58 xen-02 ha-lizard:  This iteration is count 4440
Dec 16 09:30:58 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:30:58 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:30:58 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:30:58 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:30:58 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:30:58 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:30:58 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:30:59 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:30:59 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:30:59 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:30:59 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:30:59 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:30:59 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:30:59 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:30:59 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:30:59 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:30:59 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:31:00 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:31:00 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:31:00 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:31:00 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:31:00 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:31:00 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:31:00 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:31:00 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:31:00 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:31:01 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:31:01 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:31:01 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:31:01 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:31:01 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:31:01 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:31:01 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:31:01 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:31:01 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:31:02 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:31:04 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:31:06 xen-02 iscsi-ha: 21977 Spawning new instance of iscsi-ha
Dec 16 09:31:06 xen-02 iscsi-ha: 21977 check_logger_processes Checking logger processes
Dec 16 09:31:06 xen-02 iscsi-ha: 21977 check_logger_processes No processes to clear
Dec 16 09:31:06 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:31:06 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:31:06 xen-02 iscsi-ha:  This iteration is count 6623
Dec 16 09:31:06 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:31:06 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:31:06 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:31:06 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:31:06 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:31:06 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:31:06 xen-02 iscsi-ha: 23398 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:31:06 xen-02 iscsi-ha: 23398 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:31:06 xen-02 iscsi-ha: 23398 service_execute: systemctl mode being used
Dec 16 09:31:06 xen-02 iscsi-ha: 23398 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:31:06 xen-02 iscsi-ha: 23398 service_execute: Returning exit status [ 3 ]
Dec 16 09:31:06 xen-02 iscsi-ha: 23398 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:31:06 xen-02 iscsi-ha: 23398 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485436 dw:1087485436 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:31:06 xen-02 iscsi-ha: 23398 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:31:06 xen-02 iscsi-ha: 23398 DRBD Resource: iscsi1 in Connected state
Dec 16 09:31:06 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:31:06 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:31:07 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:31:12 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:31:13 xen-02 ha-lizard: 22567 Spawning new instance of ha-lizard
Dec 16 09:31:13 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:31:13 xen-02 ha-lizard:  This iteration is count 4441
Dec 16 09:31:13 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:31:13 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:31:13 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:31:13 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:31:13 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:31:13 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:31:13 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:31:14 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:31:14 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:31:14 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:31:14 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:31:14 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:31:14 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:31:14 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:31:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:31:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:31:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:31:15 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:31:15 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:31:15 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:31:15 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:31:15 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:31:15 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:31:15 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:31:15 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:31:15 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:31:16 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:31:16 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:31:16 xen-02 iscsi-ha: 23302 Spawning new instance of iscsi-ha
Dec 16 09:31:16 xen-02 iscsi-ha: 23302 check_logger_processes Checking logger processes
Dec 16 09:31:16 xen-02 iscsi-ha: 23302 check_logger_processes No processes to clear
Dec 16 09:31:16 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:31:16 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:31:16 xen-02 iscsi-ha:  This iteration is count 6624
Dec 16 09:31:16 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:31:16 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:31:16 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:31:16 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:31:16 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:31:16 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:31:16 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:31:16 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:31:16 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:31:16 xen-02 iscsi-ha: 23842 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:31:16 xen-02 iscsi-ha: 23842 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:31:16 xen-02 iscsi-ha: 23842 service_execute: systemctl mode being used
Dec 16 09:31:16 xen-02 iscsi-ha: 23842 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:31:16 xen-02 iscsi-ha: 23842 service_execute: Returning exit status [ 3 ]
Dec 16 09:31:16 xen-02 iscsi-ha: 23842 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:31:16 xen-02 iscsi-ha: 23842 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485452 dw:1087485452 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:31:16 xen-02 iscsi-ha: 23842 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:31:16 xen-02 iscsi-ha: 23842 DRBD Resource: iscsi1 in Connected state
Dec 16 09:31:16 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:31:16 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:31:16 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:31:16 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:31:16 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:31:16 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:31:17 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:31:17 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:31:22 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:31:24 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:31:26 xen-02 iscsi-ha: 23780 Spawning new instance of iscsi-ha
Dec 16 09:31:26 xen-02 iscsi-ha: 23780 check_logger_processes Checking logger processes
Dec 16 09:31:26 xen-02 iscsi-ha: 23780 check_logger_processes No processes to clear
Dec 16 09:31:26 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:31:26 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:31:26 xen-02 iscsi-ha:  This iteration is count 6625
Dec 16 09:31:26 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:31:26 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:31:26 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:31:26 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:31:26 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:31:26 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:31:26 xen-02 iscsi-ha: 24146 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:31:26 xen-02 iscsi-ha: 24146 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:31:26 xen-02 iscsi-ha: 24146 service_execute: systemctl mode being used
Dec 16 09:31:26 xen-02 iscsi-ha: 24146 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:31:26 xen-02 iscsi-ha: 24146 service_execute: Returning exit status [ 3 ]
Dec 16 09:31:26 xen-02 iscsi-ha: 24146 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:31:26 xen-02 iscsi-ha: 24146 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485452 dw:1087485452 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:31:26 xen-02 iscsi-ha: 24146 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:31:26 xen-02 iscsi-ha: 24146 DRBD Resource: iscsi1 in Connected state
Dec 16 09:31:26 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:31:26 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:31:27 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:31:28 xen-02 ha-lizard: 23488 Spawning new instance of ha-lizard
Dec 16 09:31:28 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:31:28 xen-02 ha-lizard:  This iteration is count 4442
Dec 16 09:31:28 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:31:28 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:31:28 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:31:28 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:31:28 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:31:28 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:31:28 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:31:29 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:31:30 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:31:30 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:31:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:31:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:31:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:31:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:31:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:31:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:31:30 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:31:30 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:31:30 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:31:30 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:31:30 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:31:30 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:31:30 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:31:30 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:31:30 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:31:31 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:31:31 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:31:31 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:31:31 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:31:31 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:31:31 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:31:32 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:31:32 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:31:32 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:31:32 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:31:32 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:31:34 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:31:36 xen-02 iscsi-ha: 24111 Spawning new instance of iscsi-ha
Dec 16 09:31:36 xen-02 iscsi-ha: 24111 check_logger_processes Checking logger processes
Dec 16 09:31:36 xen-02 iscsi-ha: 24111 check_logger_processes No processes to clear
Dec 16 09:31:36 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:31:36 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:31:36 xen-02 iscsi-ha:  This iteration is count 6626
Dec 16 09:31:36 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:31:36 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:31:36 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:31:36 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:31:36 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:31:36 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:31:36 xen-02 iscsi-ha: 25763 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:31:36 xen-02 iscsi-ha: 25763 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:31:36 xen-02 iscsi-ha: 25763 service_execute: systemctl mode being used
Dec 16 09:31:36 xen-02 iscsi-ha: 25763 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:31:36 xen-02 iscsi-ha: 25763 service_execute: Returning exit status [ 3 ]
Dec 16 09:31:36 xen-02 iscsi-ha: 25763 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:31:36 xen-02 iscsi-ha: 25763 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485452 dw:1087485452 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:31:37 xen-02 iscsi-ha: 25763 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:31:37 xen-02 iscsi-ha: 25763 DRBD Resource: iscsi1 in Connected state
Dec 16 09:31:37 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:31:37 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:31:37 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:31:42 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:31:43 xen-02 ha-lizard: 24219 Spawning new instance of ha-lizard
Dec 16 09:31:43 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:31:43 xen-02 ha-lizard:  This iteration is count 4443
Dec 16 09:31:43 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:31:43 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:31:44 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:31:44 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:31:44 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:31:44 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:31:44 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:31:44 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:31:45 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:31:45 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:31:45 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:31:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:31:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:31:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:31:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:31:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:31:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:31:45 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:31:45 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:31:45 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:31:45 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:31:45 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:31:45 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:31:45 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:31:45 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:31:45 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:31:46 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:31:46 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:31:46 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:31:46 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:31:46 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:31:46 xen-02 iscsi-ha: 25728 Spawning new instance of iscsi-ha
Dec 16 09:31:46 xen-02 iscsi-ha: 25728 check_logger_processes Checking logger processes
Dec 16 09:31:46 xen-02 iscsi-ha: 25728 check_logger_processes No processes to clear
Dec 16 09:31:46 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:31:46 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:31:46 xen-02 iscsi-ha:  This iteration is count 6627
Dec 16 09:31:46 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:31:46 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:31:46 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:31:46 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:31:46 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:31:47 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:31:47 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:31:47 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:31:47 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:31:47 xen-02 iscsi-ha: 26449 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:31:47 xen-02 iscsi-ha: 26449 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:31:47 xen-02 iscsi-ha: 26449 service_execute: systemctl mode being used
Dec 16 09:31:47 xen-02 iscsi-ha: 26449 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:31:47 xen-02 iscsi-ha: 26449 service_execute: Returning exit status [ 3 ]
Dec 16 09:31:47 xen-02 iscsi-ha: 26449 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:31:47 xen-02 iscsi-ha: 26449 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485460 dw:1087485460 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:31:47 xen-02 iscsi-ha: 26449 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:31:47 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:31:47 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:31:47 xen-02 iscsi-ha: 26449 DRBD Resource: iscsi1 in Connected state
Dec 16 09:31:47 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:31:47 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:31:47 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:31:52 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:31:54 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:31:56 xen-02 iscsi-ha: 26383 Spawning new instance of iscsi-ha
Dec 16 09:31:56 xen-02 iscsi-ha: 26383 check_logger_processes Checking logger processes
Dec 16 09:31:57 xen-02 iscsi-ha: 26383 check_logger_processes No processes to clear
Dec 16 09:31:57 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:31:57 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:31:57 xen-02 iscsi-ha:  This iteration is count 6628
Dec 16 09:31:57 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:31:57 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:31:57 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:31:57 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:31:57 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:31:57 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:31:57 xen-02 iscsi-ha: 26709 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:31:57 xen-02 iscsi-ha: 26709 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:31:57 xen-02 iscsi-ha: 26709 service_execute: systemctl mode being used
Dec 16 09:31:57 xen-02 iscsi-ha: 26709 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:31:57 xen-02 iscsi-ha: 26709 service_execute: Returning exit status [ 3 ]
Dec 16 09:31:57 xen-02 iscsi-ha: 26709 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:31:57 xen-02 iscsi-ha: 26709 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485460 dw:1087485460 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:31:57 xen-02 iscsi-ha: 26709 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:31:57 xen-02 iscsi-ha: 26709 DRBD Resource: iscsi1 in Connected state
Dec 16 09:31:57 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:31:57 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:31:57 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:31:58 xen-02 ha-lizard: 26041 Spawning new instance of ha-lizard
Dec 16 09:31:58 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:31:58 xen-02 ha-lizard:  This iteration is count 4444
Dec 16 09:31:58 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:31:58 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:31:59 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:31:59 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:31:59 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:31:59 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:31:59 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:32:00 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:32:00 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:32:00 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:32:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:32:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:32:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:32:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:32:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:32:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:32:00 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:32:00 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:32:00 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:32:00 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:32:00 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:32:00 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:32:00 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:32:00 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:32:00 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:32:01 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:32:01 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:32:01 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:32:01 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:32:01 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:32:02 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:32:02 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:32:02 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:32:02 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:32:02 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:32:02 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:04 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:32:07 xen-02 iscsi-ha: 26674 Spawning new instance of iscsi-ha
Dec 16 09:32:07 xen-02 iscsi-ha: 26674 check_logger_processes Checking logger processes
Dec 16 09:32:07 xen-02 iscsi-ha: 26674 check_logger_processes No processes to clear
Dec 16 09:32:07 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:32:07 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:32:07 xen-02 iscsi-ha:  This iteration is count 6629
Dec 16 09:32:07 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:07 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:32:07 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:32:07 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:32:07 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:32:07 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:32:07 xen-02 iscsi-ha: 27964 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:32:07 xen-02 iscsi-ha: 27964 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:32:07 xen-02 iscsi-ha: 27964 service_execute: systemctl mode being used
Dec 16 09:32:07 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:07 xen-02 iscsi-ha: 27964 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:32:07 xen-02 iscsi-ha: 27964 service_execute: Returning exit status [ 3 ]
Dec 16 09:32:07 xen-02 iscsi-ha: 27964 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:32:07 xen-02 iscsi-ha: 27964 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485460 dw:1087485460 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:32:07 xen-02 iscsi-ha: 27964 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:32:07 xen-02 iscsi-ha: 27964 DRBD Resource: iscsi1 in Connected state
Dec 16 09:32:07 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:07 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:12 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:13 xen-02 ha-lizard: 26782 Spawning new instance of ha-lizard
Dec 16 09:32:14 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:32:14 xen-02 ha-lizard:  This iteration is count 4445
Dec 16 09:32:14 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:14 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:32:14 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:32:14 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:32:14 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:32:14 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:32:14 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:32:14 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:32:15 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:32:15 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:32:15 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:32:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:32:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:32:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:32:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:32:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:32:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:32:15 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:32:15 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:32:15 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:32:15 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:32:15 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:32:15 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:32:15 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:32:16 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:32:16 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:32:16 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:32:16 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:32:16 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:32:16 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:32:16 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:32:17 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:32:17 xen-02 iscsi-ha: 27890 Spawning new instance of iscsi-ha
Dec 16 09:32:17 xen-02 iscsi-ha: 27890 check_logger_processes Checking logger processes
Dec 16 09:32:17 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:32:17 xen-02 iscsi-ha: 27890 check_logger_processes No processes to clear
Dec 16 09:32:17 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:32:17 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:32:17 xen-02 iscsi-ha:  This iteration is count 6630
Dec 16 09:32:17 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:17 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:32:17 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:32:17 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:32:17 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:32:17 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:17 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:32:17 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:32:17 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:32:17 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:32:17 xen-02 iscsi-ha: 29042 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:32:17 xen-02 iscsi-ha: 29042 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:32:17 xen-02 iscsi-ha: 29042 service_execute: systemctl mode being used
Dec 16 09:32:17 xen-02 iscsi-ha: 29042 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:32:17 xen-02 iscsi-ha: 29042 service_execute: Returning exit status [ 3 ]
Dec 16 09:32:17 xen-02 iscsi-ha: 29042 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:32:17 xen-02 iscsi-ha: 29042 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485476 dw:1087485476 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:32:17 xen-02 iscsi-ha: 29042 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:32:17 xen-02 iscsi-ha: 29042 DRBD Resource: iscsi1 in Connected state
Dec 16 09:32:17 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:17 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:22 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:24 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:32:27 xen-02 iscsi-ha: 28978 Spawning new instance of iscsi-ha
Dec 16 09:32:27 xen-02 iscsi-ha: 28978 check_logger_processes Checking logger processes
Dec 16 09:32:27 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:27 xen-02 iscsi-ha: 28978 check_logger_processes No processes to clear
Dec 16 09:32:27 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:32:27 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:32:27 xen-02 iscsi-ha:  This iteration is count 6631
Dec 16 09:32:27 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:27 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:32:27 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:32:27 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:32:27 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:32:27 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:32:27 xen-02 iscsi-ha: 29268 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:32:27 xen-02 iscsi-ha: 29268 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:32:27 xen-02 iscsi-ha: 29268 service_execute: systemctl mode being used
Dec 16 09:32:27 xen-02 iscsi-ha: 29268 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:32:27 xen-02 iscsi-ha: 29268 service_execute: Returning exit status [ 3 ]
Dec 16 09:32:27 xen-02 iscsi-ha: 29268 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:32:27 xen-02 iscsi-ha: 29268 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485476 dw:1087485476 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:32:27 xen-02 iscsi-ha: 29268 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:32:27 xen-02 iscsi-ha: 29268 DRBD Resource: iscsi1 in Connected state
Dec 16 09:32:27 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:27 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:29 xen-02 ha-lizard: 28608 Spawning new instance of ha-lizard
Dec 16 09:32:29 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:32:29 xen-02 ha-lizard:  This iteration is count 4446
Dec 16 09:32:29 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:29 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:32:29 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:32:29 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:32:29 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:32:29 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:32:29 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:32:30 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:32:30 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:32:30 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:32:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:32:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:32:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:32:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:32:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:32:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:32:30 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:32:30 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:32:30 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:32:30 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:32:31 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:32:31 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:32:31 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:32:31 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:32:31 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:32:31 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:32:31 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:32:32 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:32:32 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:32:32 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:32:32 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:32:32 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:32:32 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:32 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:32:32 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:32:32 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:32:35 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:32:37 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:37 xen-02 iscsi-ha: 29233 Spawning new instance of iscsi-ha
Dec 16 09:32:37 xen-02 iscsi-ha: 29233 check_logger_processes Checking logger processes
Dec 16 09:32:37 xen-02 iscsi-ha: 29233 check_logger_processes No processes to clear
Dec 16 09:32:37 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:32:37 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:32:37 xen-02 iscsi-ha:  This iteration is count 6632
Dec 16 09:32:37 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:37 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:32:37 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:32:37 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:32:37 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:32:37 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:32:37 xen-02 iscsi-ha: 29867 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:32:37 xen-02 iscsi-ha: 29867 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:32:37 xen-02 iscsi-ha: 29867 service_execute: systemctl mode being used
Dec 16 09:32:37 xen-02 iscsi-ha: 29867 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:32:37 xen-02 iscsi-ha: 29867 service_execute: Returning exit status [ 3 ]
Dec 16 09:32:37 xen-02 iscsi-ha: 29867 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:32:37 xen-02 iscsi-ha: 29867 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485476 dw:1087485476 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:32:37 xen-02 iscsi-ha: 29867 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:32:37 xen-02 iscsi-ha: 29867 DRBD Resource: iscsi1 in Connected state
Dec 16 09:32:38 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:38 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:42 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:44 xen-02 ha-lizard: 29339 Spawning new instance of ha-lizard
Dec 16 09:32:44 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:32:44 xen-02 ha-lizard:  This iteration is count 4447
Dec 16 09:32:44 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:44 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:32:44 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:32:44 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:32:44 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:32:44 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:32:44 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:32:45 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:32:45 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:32:45 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:32:45 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:32:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:32:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:32:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:32:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:32:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:32:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:32:45 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:32:45 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:32:45 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:32:45 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:32:46 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:32:46 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:32:46 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:32:46 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:32:46 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:32:46 xen-02 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:32:46 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:32:46 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:32:46 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:32:46 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:32:47 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:32:47 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:32:47 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:32:47 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:32:47 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:32:47 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:47 xen-02 iscsi-ha: 29832 Spawning new instance of iscsi-ha
Dec 16 09:32:47 xen-02 iscsi-ha: 29832 check_logger_processes Checking logger processes
Dec 16 09:32:47 xen-02 iscsi-ha: 29832 check_logger_processes No processes to clear
Dec 16 09:32:47 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:32:47 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:32:47 xen-02 iscsi-ha:  This iteration is count 6633
Dec 16 09:32:47 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:47 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:32:47 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:32:47 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:32:48 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:32:48 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:32:48 xen-02 iscsi-ha: 31675 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:32:48 xen-02 iscsi-ha: 31675 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:32:48 xen-02 iscsi-ha: 31675 service_execute: systemctl mode being used
Dec 16 09:32:48 xen-02 iscsi-ha: 31675 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:32:48 xen-02 iscsi-ha: 31675 service_execute: Returning exit status [ 3 ]
Dec 16 09:32:48 xen-02 iscsi-ha: 31675 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:32:48 xen-02 iscsi-ha: 31675 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485484 dw:1087485484 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:32:48 xen-02 iscsi-ha: 31675 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:32:48 xen-02 iscsi-ha: 31675 DRBD Resource: iscsi1 in Connected state
Dec 16 09:32:48 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:48 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:52 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:55 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:32:57 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:57 xen-02 iscsi-ha: 31621 Spawning new instance of iscsi-ha
Dec 16 09:32:57 xen-02 iscsi-ha: 31621 check_logger_processes Checking logger processes
Dec 16 09:32:57 xen-02 iscsi-ha: 31621 check_logger_processes No processes to clear
Dec 16 09:32:58 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:32:58 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:32:58 xen-02 iscsi-ha:  This iteration is count 6634
Dec 16 09:32:58 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:58 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:32:58 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:32:58 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:32:58 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:32:58 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:32:58 xen-02 iscsi-ha: 31827 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:32:58 xen-02 iscsi-ha: 31827 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:32:58 xen-02 iscsi-ha: 31827 service_execute: systemctl mode being used
Dec 16 09:32:58 xen-02 iscsi-ha: 31827 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:32:58 xen-02 iscsi-ha: 31827 service_execute: Returning exit status [ 3 ]
Dec 16 09:32:58 xen-02 iscsi-ha: 31827 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:32:58 xen-02 iscsi-ha: 31827 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485484 dw:1087485484 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:32:58 xen-02 iscsi-ha: 31827 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:32:58 xen-02 iscsi-ha: 31827 DRBD Resource: iscsi1 in Connected state
Dec 16 09:32:58 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:58 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:59 xen-02 ha-lizard: 31135 Spawning new instance of ha-lizard
Dec 16 09:32:59 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:32:59 xen-02 ha-lizard:  This iteration is count 4448
Dec 16 09:32:59 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:59 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:32:59 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:32:59 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:32:59 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:32:59 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:32:59 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:33:00 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:33:00 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:33:00 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:33:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:33:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:33:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:33:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:33:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:33:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:33:00 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:33:01 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:33:01 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:33:01 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:33:01 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:33:01 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:33:01 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:33:01 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:33:01 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:33:01 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:33:01 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:33:02 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:33:02 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:33:02 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:33:02 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:33:02 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:33:02 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:33:02 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:02 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:33:02 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:33:05 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:33:07 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:08 xen-02 iscsi-ha: 31792 Spawning new instance of iscsi-ha
Dec 16 09:33:08 xen-02 iscsi-ha: 31792 check_logger_processes Checking logger processes
Dec 16 09:33:08 xen-02 iscsi-ha: 31792 check_logger_processes No processes to clear
Dec 16 09:33:08 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:33:08 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:33:08 xen-02 iscsi-ha:  This iteration is count 6635
Dec 16 09:33:08 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:08 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:33:08 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:33:08 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:33:08 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:33:08 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:33:08 xen-02 iscsi-ha: 32421 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:33:08 xen-02 iscsi-ha: 32421 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:33:08 xen-02 iscsi-ha: 32421 service_execute: systemctl mode being used
Dec 16 09:33:08 xen-02 iscsi-ha: 32421 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:33:08 xen-02 iscsi-ha: 32421 service_execute: Returning exit status [ 3 ]
Dec 16 09:33:08 xen-02 iscsi-ha: 32421 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:33:08 xen-02 iscsi-ha: 32421 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485484 dw:1087485484 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:33:08 xen-02 iscsi-ha: 32421 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:33:08 xen-02 iscsi-ha: 32421 DRBD Resource: iscsi1 in Connected state
Dec 16 09:33:08 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:08 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:12 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:14 xen-02 ha-lizard: 31894 Spawning new instance of ha-lizard
Dec 16 09:33:14 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:33:14 xen-02 ha-lizard:  This iteration is count 4449
Dec 16 09:33:14 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:14 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:33:14 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:33:14 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:33:14 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:33:14 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:33:14 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:33:15 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:33:15 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:33:15 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:33:15 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:33:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:33:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:33:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:33:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:33:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:33:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:33:16 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:33:16 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:33:16 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:33:16 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:33:16 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:33:16 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:33:16 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:33:16 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:33:16 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:33:16 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:33:16 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:33:17 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:33:17 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:33:17 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:33:17 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:33:17 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:33:17 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:17 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:33:17 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:33:17 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:33:18 xen-02 iscsi-ha: 32386 Spawning new instance of iscsi-ha
Dec 16 09:33:18 xen-02 iscsi-ha: 32386 check_logger_processes Checking logger processes
Dec 16 09:33:18 xen-02 iscsi-ha: 32386 check_logger_processes No processes to clear
Dec 16 09:33:18 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:33:18 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:33:18 xen-02 iscsi-ha:  This iteration is count 6636
Dec 16 09:33:18 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:18 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:33:18 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:33:18 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:33:18 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:33:18 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:33:18 xen-02 iscsi-ha: 1790 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:33:18 xen-02 iscsi-ha: 1790 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:33:18 xen-02 iscsi-ha: 1790 service_execute: systemctl mode being used
Dec 16 09:33:18 xen-02 iscsi-ha: 1790 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:33:18 xen-02 iscsi-ha: 1790 service_execute: Returning exit status [ 3 ]
Dec 16 09:33:18 xen-02 iscsi-ha: 1790 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:33:18 xen-02 iscsi-ha: 1790 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485496 dw:1087485496 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:33:18 xen-02 iscsi-ha: 1790 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:33:18 xen-02 iscsi-ha: 1790 DRBD Resource: iscsi1 in Connected state
Dec 16 09:33:18 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:18 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:22 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:25 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:33:27 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:28 xen-02 iscsi-ha: 1715 Spawning new instance of iscsi-ha
Dec 16 09:33:28 xen-02 iscsi-ha: 1715 check_logger_processes Checking logger processes
Dec 16 09:33:28 xen-02 iscsi-ha: 1715 check_logger_processes No processes to clear
Dec 16 09:33:28 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:33:28 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:33:28 xen-02 iscsi-ha:  This iteration is count 6637
Dec 16 09:33:28 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:28 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:33:28 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:33:28 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:33:28 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:33:28 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:33:28 xen-02 iscsi-ha: 2239 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:33:28 xen-02 iscsi-ha: 2239 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:33:28 xen-02 iscsi-ha: 2239 service_execute: systemctl mode being used
Dec 16 09:33:28 xen-02 iscsi-ha: 2239 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:33:28 xen-02 iscsi-ha: 2239 service_execute: Returning exit status [ 3 ]
Dec 16 09:33:28 xen-02 iscsi-ha: 2239 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:33:28 xen-02 iscsi-ha: 2239 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485496 dw:1087485496 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:33:28 xen-02 iscsi-ha: 2239 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:33:28 xen-02 iscsi-ha: 2239 DRBD Resource: iscsi1 in Connected state
Dec 16 09:33:28 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:28 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:29 xen-02 ha-lizard: 32689 Spawning new instance of ha-lizard
Dec 16 09:33:29 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:33:29 xen-02 ha-lizard:  This iteration is count 4450
Dec 16 09:33:29 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:29 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:33:29 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:33:29 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:33:29 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:33:29 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:33:29 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:33:30 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:33:31 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:33:31 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:33:31 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:33:31 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:33:31 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:33:31 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:33:31 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:33:31 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:33:31 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:33:31 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:33:31 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:33:31 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:33:31 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:33:31 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:33:31 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:33:31 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:33:31 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:33:31 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:33:31 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:33:32 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:33:32 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:33:32 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:33:32 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:33:32 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:33:32 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:32 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:33:32 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:33:32 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:33:35 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:33:37 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:38 xen-02 iscsi-ha: 2200 Spawning new instance of iscsi-ha
Dec 16 09:33:38 xen-02 iscsi-ha: 2200 check_logger_processes Checking logger processes
Dec 16 09:33:38 xen-02 iscsi-ha: 2200 check_logger_processes No processes to clear
Dec 16 09:33:38 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:33:38 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:33:38 xen-02 iscsi-ha:  This iteration is count 6638
Dec 16 09:33:38 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:38 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:33:38 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:33:38 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:33:38 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:33:38 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:33:38 xen-02 iscsi-ha: 2849 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:33:38 xen-02 iscsi-ha: 2849 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:33:38 xen-02 iscsi-ha: 2849 service_execute: systemctl mode being used
Dec 16 09:33:38 xen-02 iscsi-ha: 2849 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:33:38 xen-02 iscsi-ha: 2849 service_execute: Returning exit status [ 3 ]
Dec 16 09:33:38 xen-02 iscsi-ha: 2849 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:33:38 xen-02 iscsi-ha: 2849 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485496 dw:1087485496 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:33:38 xen-02 iscsi-ha: 2849 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:33:38 xen-02 iscsi-ha: 2849 DRBD Resource: iscsi1 in Connected state
Dec 16 09:33:39 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:39 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:42 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:44 xen-02 ha-lizard: 2312 Spawning new instance of ha-lizard
Dec 16 09:33:44 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:33:44 xen-02 ha-lizard:  This iteration is count 4451
Dec 16 09:33:44 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:44 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:33:45 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:33:45 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:33:45 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:33:45 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:33:45 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:33:45 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:33:46 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:33:46 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:33:46 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:33:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:33:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:33:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:33:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:33:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:33:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:33:46 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:33:46 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:33:46 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:33:46 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:33:46 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:33:46 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:33:46 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:33:46 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:33:46 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:33:47 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:33:47 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:33:47 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:33:47 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:33:47 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:33:47 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:47 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:33:47 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:33:47 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:33:47 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:33:47 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:33:48 xen-02 iscsi-ha: 2814 Spawning new instance of iscsi-ha
Dec 16 09:33:48 xen-02 iscsi-ha: 2814 check_logger_processes Checking logger processes
Dec 16 09:33:48 xen-02 iscsi-ha: 2814 check_logger_processes No processes to clear
Dec 16 09:33:48 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:33:48 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:33:48 xen-02 iscsi-ha:  This iteration is count 6639
Dec 16 09:33:48 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:48 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:33:48 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:33:48 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:33:48 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:33:48 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:33:48 xen-02 iscsi-ha: 3663 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:33:48 xen-02 iscsi-ha: 3663 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:33:48 xen-02 iscsi-ha: 3663 service_execute: systemctl mode being used
Dec 16 09:33:48 xen-02 iscsi-ha: 3663 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:33:48 xen-02 iscsi-ha: 3663 service_execute: Returning exit status [ 3 ]
Dec 16 09:33:49 xen-02 iscsi-ha: 3663 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:33:49 xen-02 iscsi-ha: 3663 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485496 dw:1087485496 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:33:49 xen-02 iscsi-ha: 3663 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:33:49 xen-02 iscsi-ha: 3663 DRBD Resource: iscsi1 in Connected state
Dec 16 09:33:49 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:49 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:52 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:55 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:33:57 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:58 xen-02 iscsi-ha: 3628 Spawning new instance of iscsi-ha
Dec 16 09:33:58 xen-02 iscsi-ha: 3628 check_logger_processes Checking logger processes
Dec 16 09:33:58 xen-02 iscsi-ha: 3628 check_logger_processes No processes to clear
Dec 16 09:33:58 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:33:58 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:33:58 xen-02 iscsi-ha:  This iteration is count 6640
Dec 16 09:33:58 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:58 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:33:58 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:33:58 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:33:59 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:33:59 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:33:59 xen-02 iscsi-ha: 5010 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:33:59 xen-02 iscsi-ha: 5010 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:33:59 xen-02 iscsi-ha: 5010 service_execute: systemctl mode being used
Dec 16 09:33:59 xen-02 iscsi-ha: 5010 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:33:59 xen-02 iscsi-ha: 5010 service_execute: Returning exit status [ 3 ]
Dec 16 09:33:59 xen-02 iscsi-ha: 5010 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:33:59 xen-02 iscsi-ha: 5010 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485500 dw:1087485500 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:33:59 xen-02 iscsi-ha: 5010 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:33:59 xen-02 iscsi-ha: 5010 DRBD Resource: iscsi1 in Connected state
Dec 16 09:33:59 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:59 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:59 xen-02 ha-lizard: 3127 Spawning new instance of ha-lizard
Dec 16 09:33:59 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:33:59 xen-02 ha-lizard:  This iteration is count 4452
Dec 16 09:33:59 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:59 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:34:00 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:34:00 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:34:00 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:34:00 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:34:00 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:34:01 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:34:01 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:34:01 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:34:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:34:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:34:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:34:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:34:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:34:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:34:01 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:34:01 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:34:01 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:34:01 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:34:01 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:34:01 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:34:01 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:34:01 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:34:01 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:34:02 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:34:02 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:34:02 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:34:02 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:34:02 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:34:02 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:02 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:34:02 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:34:02 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:34:03 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:34:03 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:34:05 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:34:07 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:09 xen-02 iscsi-ha: 4975 Spawning new instance of iscsi-ha
Dec 16 09:34:09 xen-02 iscsi-ha: 4975 check_logger_processes Checking logger processes
Dec 16 09:34:09 xen-02 iscsi-ha: 4975 check_logger_processes No processes to clear
Dec 16 09:34:09 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:34:09 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:34:09 xen-02 iscsi-ha:  This iteration is count 6641
Dec 16 09:34:09 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:09 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:34:09 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:34:09 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:34:09 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:34:09 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:34:09 xen-02 iscsi-ha: 5616 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:34:09 xen-02 iscsi-ha: 5616 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:34:09 xen-02 iscsi-ha: 5616 service_execute: systemctl mode being used
Dec 16 09:34:09 xen-02 iscsi-ha: 5616 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:34:09 xen-02 iscsi-ha: 5616 service_execute: Returning exit status [ 3 ]
Dec 16 09:34:09 xen-02 iscsi-ha: 5616 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:34:09 xen-02 iscsi-ha: 5616 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485500 dw:1087485500 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:34:09 xen-02 iscsi-ha: 5616 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:34:09 xen-02 iscsi-ha: 5616 DRBD Resource: iscsi1 in Connected state
Dec 16 09:34:09 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:09 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:12 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:14 xen-02 ha-lizard: 5077 Spawning new instance of ha-lizard
Dec 16 09:34:15 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:34:15 xen-02 ha-lizard:  This iteration is count 4453
Dec 16 09:34:15 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:15 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:34:15 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:34:15 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:34:15 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:34:15 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:34:15 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:34:15 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:34:16 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:34:16 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:34:16 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:34:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:34:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:34:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:34:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:34:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:34:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:34:16 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:34:16 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:34:16 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:34:16 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:34:16 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:34:16 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:34:16 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:34:16 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:34:17 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:34:17 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:34:17 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:34:17 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:34:17 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:34:17 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:34:17 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:17 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:34:18 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:34:18 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:34:18 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:34:18 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:34:19 xen-02 iscsi-ha: 5578 Spawning new instance of iscsi-ha
Dec 16 09:34:19 xen-02 iscsi-ha: 5578 check_logger_processes Checking logger processes
Dec 16 09:34:19 xen-02 iscsi-ha: 5578 check_logger_processes No processes to clear
Dec 16 09:34:19 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:34:19 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:34:19 xen-02 iscsi-ha:  This iteration is count 6642
Dec 16 09:34:19 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:19 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:34:19 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:34:19 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:34:19 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:34:19 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:34:19 xen-02 iscsi-ha: 6424 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:34:19 xen-02 iscsi-ha: 6424 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:34:19 xen-02 iscsi-ha: 6424 service_execute: systemctl mode being used
Dec 16 09:34:19 xen-02 iscsi-ha: 6424 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:34:19 xen-02 iscsi-ha: 6424 service_execute: Returning exit status [ 3 ]
Dec 16 09:34:19 xen-02 iscsi-ha: 6424 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:34:19 xen-02 iscsi-ha: 6424 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485500 dw:1087485500 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:34:19 xen-02 iscsi-ha: 6424 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:34:19 xen-02 iscsi-ha: 6424 DRBD Resource: iscsi1 in Connected state
Dec 16 09:34:19 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:19 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:22 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:25 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:34:27 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:29 xen-02 iscsi-ha: 6389 Spawning new instance of iscsi-ha
Dec 16 09:34:29 xen-02 iscsi-ha: 6389 check_logger_processes Checking logger processes
Dec 16 09:34:29 xen-02 iscsi-ha: 6389 check_logger_processes No processes to clear
Dec 16 09:34:29 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:34:29 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:34:29 xen-02 iscsi-ha:  This iteration is count 6643
Dec 16 09:34:29 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:29 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:34:29 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:34:29 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:34:29 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:34:29 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:34:29 xen-02 iscsi-ha: 7596 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:34:29 xen-02 iscsi-ha: 7596 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:34:29 xen-02 iscsi-ha: 7596 service_execute: systemctl mode being used
Dec 16 09:34:29 xen-02 iscsi-ha: 7596 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:34:29 xen-02 iscsi-ha: 7596 service_execute: Returning exit status [ 3 ]
Dec 16 09:34:29 xen-02 iscsi-ha: 7596 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:34:29 xen-02 iscsi-ha: 7596 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485508 dw:1087485508 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:34:29 xen-02 iscsi-ha: 7596 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:34:29 xen-02 iscsi-ha: 7596 DRBD Resource: iscsi1 in Connected state
Dec 16 09:34:29 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:29 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:30 xen-02 ha-lizard: 5742 Spawning new instance of ha-lizard
Dec 16 09:34:30 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:34:30 xen-02 ha-lizard:  This iteration is count 4454
Dec 16 09:34:30 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:30 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:34:30 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:34:30 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:34:30 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:34:30 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:34:30 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:34:31 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:34:31 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:34:31 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:34:31 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:34:31 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:34:31 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:34:31 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:34:31 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:34:31 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:34:31 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:34:31 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:34:31 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:34:31 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:34:32 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:34:32 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:34:32 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:34:32 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:34:32 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:34:32 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:34:32 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:34:32 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:32 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:34:32 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:34:32 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:34:33 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:34:33 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:34:33 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:34:33 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:34:33 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:34:35 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:34:37 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:39 xen-02 iscsi-ha: 7536 Spawning new instance of iscsi-ha
Dec 16 09:34:39 xen-02 iscsi-ha: 7536 check_logger_processes Checking logger processes
Dec 16 09:34:39 xen-02 iscsi-ha: 7536 check_logger_processes No processes to clear
Dec 16 09:34:39 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:34:39 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:34:39 xen-02 iscsi-ha:  This iteration is count 6644
Dec 16 09:34:39 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:39 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:34:39 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:34:39 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:34:39 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:34:39 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:34:39 xen-02 iscsi-ha: 8211 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:34:39 xen-02 iscsi-ha: 8211 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:34:39 xen-02 iscsi-ha: 8211 service_execute: systemctl mode being used
Dec 16 09:34:39 xen-02 iscsi-ha: 8211 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:34:39 xen-02 iscsi-ha: 8211 service_execute: Returning exit status [ 3 ]
Dec 16 09:34:39 xen-02 iscsi-ha: 8211 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:34:39 xen-02 iscsi-ha: 8211 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485508 dw:1087485508 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:34:39 xen-02 iscsi-ha: 8211 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:34:39 xen-02 iscsi-ha: 8211 DRBD Resource: iscsi1 in Connected state
Dec 16 09:34:39 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:39 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:42 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:45 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:34:45 xen-02 ha-lizard: 7668 Spawning new instance of ha-lizard
Dec 16 09:34:45 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:34:45 xen-02 ha-lizard:  This iteration is count 4455
Dec 16 09:34:45 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:45 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:34:45 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:34:45 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:34:45 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:34:45 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:34:45 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:34:46 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:34:46 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:34:46 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:34:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:34:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:34:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:34:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:34:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:34:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:34:46 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:34:46 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:34:46 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:34:46 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:34:47 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:34:47 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:34:47 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:34:47 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:34:47 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:34:47 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:34:47 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:34:47 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:48 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:34:48 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:34:48 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:34:48 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:34:48 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:34:48 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:34:48 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:34:48 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:34:49 xen-02 iscsi-ha: 8176 Spawning new instance of iscsi-ha
Dec 16 09:34:49 xen-02 iscsi-ha: 8176 check_logger_processes Checking logger processes
Dec 16 09:34:49 xen-02 iscsi-ha: 8176 check_logger_processes No processes to clear
Dec 16 09:34:49 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:34:49 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:34:49 xen-02 iscsi-ha:  This iteration is count 6645
Dec 16 09:34:49 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:49 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:34:49 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:34:49 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:34:49 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:34:49 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:34:49 xen-02 iscsi-ha: 9003 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:34:49 xen-02 iscsi-ha: 9003 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:34:49 xen-02 iscsi-ha: 9003 service_execute: systemctl mode being used
Dec 16 09:34:49 xen-02 iscsi-ha: 9003 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:34:49 xen-02 iscsi-ha: 9003 service_execute: Returning exit status [ 3 ]
Dec 16 09:34:49 xen-02 iscsi-ha: 9003 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:34:49 xen-02 iscsi-ha: 9003 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485508 dw:1087485508 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:34:49 xen-02 iscsi-ha: 9003 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:34:49 xen-02 iscsi-ha: 9003 DRBD Resource: iscsi1 in Connected state
Dec 16 09:34:50 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:50 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:52 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:55 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:34:57 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:59 xen-02 iscsi-ha: 8968 Spawning new instance of iscsi-ha
Dec 16 09:34:59 xen-02 iscsi-ha: 8968 check_logger_processes Checking logger processes
Dec 16 09:34:59 xen-02 iscsi-ha: 8968 check_logger_processes No processes to clear
Dec 16 09:34:59 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:34:59 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:34:59 xen-02 iscsi-ha:  This iteration is count 6646
Dec 16 09:34:59 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:59 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:34:59 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:34:59 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:35:00 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:35:00 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:35:00 xen-02 iscsi-ha: 9227 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:35:00 xen-02 iscsi-ha: 9227 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:35:00 xen-02 iscsi-ha: 9227 service_execute: systemctl mode being used
Dec 16 09:35:00 xen-02 iscsi-ha: 9227 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:35:00 xen-02 iscsi-ha: 9227 service_execute: Returning exit status [ 3 ]
Dec 16 09:35:00 xen-02 iscsi-ha: 9227 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:35:00 xen-02 iscsi-ha: 9227 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485512 dw:1087485512 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:35:00 xen-02 iscsi-ha: 9227 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:35:00 xen-02 iscsi-ha: 9227 DRBD Resource: iscsi1 in Connected state
Dec 16 09:35:00 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:00 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:00 xen-02 ha-lizard: 8309 Spawning new instance of ha-lizard
Dec 16 09:35:00 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:35:00 xen-02 ha-lizard:  This iteration is count 4456
Dec 16 09:35:00 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:00 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:35:00 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:35:00 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:35:00 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:35:00 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:35:00 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:35:01 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:35:01 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:35:01 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:35:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:35:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:35:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:35:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:35:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:35:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:35:01 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:35:02 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:35:02 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:35:02 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:35:02 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:35:02 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:35:02 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:35:02 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:35:02 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:35:02 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:35:02 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:02 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:35:03 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:35:03 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:35:03 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:35:03 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:35:03 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:35:03 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:35:03 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:35:03 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:35:05 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:35:07 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:09 xen-02 iscsi-ha: 9148 Spawning new instance of iscsi-ha
Dec 16 09:35:09 xen-02 iscsi-ha: 9148 check_logger_processes Checking logger processes
Dec 16 09:35:09 xen-02 iscsi-ha: 9148 check_logger_processes No processes to clear
Dec 16 09:35:10 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:35:10 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:35:10 xen-02 iscsi-ha:  This iteration is count 6647
Dec 16 09:35:10 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:10 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:35:10 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:35:10 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:35:10 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:35:10 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:35:10 xen-02 iscsi-ha: 10784 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:35:10 xen-02 iscsi-ha: 10784 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:35:10 xen-02 iscsi-ha: 10784 service_execute: systemctl mode being used
Dec 16 09:35:10 xen-02 iscsi-ha: 10784 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:35:10 xen-02 iscsi-ha: 10784 service_execute: Returning exit status [ 3 ]
Dec 16 09:35:10 xen-02 iscsi-ha: 10784 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:35:10 xen-02 iscsi-ha: 10784 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485512 dw:1087485512 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:35:10 xen-02 iscsi-ha: 10784 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:35:10 xen-02 iscsi-ha: 10784 DRBD Resource: iscsi1 in Connected state
Dec 16 09:35:10 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:10 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:12 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:15 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:35:15 xen-02 ha-lizard: 9370 Spawning new instance of ha-lizard
Dec 16 09:35:15 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:35:15 xen-02 ha-lizard:  This iteration is count 4457
Dec 16 09:35:15 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:15 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:35:15 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:35:15 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:35:15 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:35:15 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:35:15 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:35:16 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:35:16 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:35:16 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:35:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:35:17 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:35:17 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:35:17 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:35:17 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:35:17 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:35:17 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:35:17 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:35:17 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:35:17 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:35:17 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:35:17 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:35:17 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:35:17 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:35:17 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:35:17 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:35:17 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:17 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:35:18 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:35:18 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:35:18 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:35:18 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:35:18 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:35:18 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:35:18 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:35:18 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:35:20 xen-02 iscsi-ha: 10747 Spawning new instance of iscsi-ha
Dec 16 09:35:20 xen-02 iscsi-ha: 10747 check_logger_processes Checking logger processes
Dec 16 09:35:20 xen-02 iscsi-ha: 10747 check_logger_processes No processes to clear
Dec 16 09:35:20 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:35:20 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:35:20 xen-02 iscsi-ha:  This iteration is count 6648
Dec 16 09:35:20 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:20 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:35:20 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:35:20 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:35:20 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:35:20 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:35:20 xen-02 iscsi-ha: 11563 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:35:20 xen-02 iscsi-ha: 11563 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:35:20 xen-02 iscsi-ha: 11563 service_execute: systemctl mode being used
Dec 16 09:35:20 xen-02 iscsi-ha: 11563 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:35:20 xen-02 iscsi-ha: 11563 service_execute: Returning exit status [ 3 ]
Dec 16 09:35:20 xen-02 iscsi-ha: 11563 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:35:20 xen-02 iscsi-ha: 11563 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485512 dw:1087485512 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:35:20 xen-02 iscsi-ha: 11563 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:35:20 xen-02 iscsi-ha: 11563 DRBD Resource: iscsi1 in Connected state
Dec 16 09:35:20 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:20 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:22 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:25 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:35:27 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:30 xen-02 iscsi-ha: 11528 Spawning new instance of iscsi-ha
Dec 16 09:35:30 xen-02 iscsi-ha: 11528 check_logger_processes Checking logger processes
Dec 16 09:35:30 xen-02 iscsi-ha: 11528 check_logger_processes No processes to clear
Dec 16 09:35:30 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:35:30 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:35:30 xen-02 iscsi-ha:  This iteration is count 6649
Dec 16 09:35:30 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:30 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:35:30 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:35:30 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:35:30 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:35:30 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:35:30 xen-02 iscsi-ha: 11961 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:35:30 xen-02 iscsi-ha: 11961 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:35:30 xen-02 iscsi-ha: 11961 service_execute: systemctl mode being used
Dec 16 09:35:30 xen-02 iscsi-ha: 11961 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:35:30 xen-02 iscsi-ha: 11961 service_execute: Returning exit status [ 3 ]
Dec 16 09:35:30 xen-02 iscsi-ha: 11961 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:35:30 xen-02 iscsi-ha: 11961 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485528 dw:1087485528 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:35:30 xen-02 iscsi-ha: 11961 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:35:30 xen-02 iscsi-ha: 11961 DRBD Resource: iscsi1 in Connected state
Dec 16 09:35:30 xen-02 ha-lizard: 10874 Spawning new instance of ha-lizard
Dec 16 09:35:30 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:30 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:30 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:35:30 xen-02 ha-lizard:  This iteration is count 4458
Dec 16 09:35:30 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:30 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:35:30 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:35:30 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:35:31 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:35:31 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:35:31 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:35:32 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:35:32 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:35:32 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:35:32 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:35:32 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:35:32 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:35:32 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:35:32 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:35:32 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:35:32 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:35:32 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:35:32 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:35:32 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:35:32 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:35:32 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:35:32 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:35:32 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:35:32 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:35:32 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:32 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:35:33 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:35:33 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:35:33 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:35:33 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:35:33 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:35:33 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:35:33 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:35:33 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:35:33 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:35:35 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:35:37 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:40 xen-02 iscsi-ha: 11926 Spawning new instance of iscsi-ha
Dec 16 09:35:40 xen-02 iscsi-ha: 11926 check_logger_processes Checking logger processes
Dec 16 09:35:40 xen-02 iscsi-ha: 11926 check_logger_processes No processes to clear
Dec 16 09:35:40 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:35:40 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:35:40 xen-02 iscsi-ha:  This iteration is count 6650
Dec 16 09:35:40 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:40 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:35:40 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:35:40 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:35:40 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:35:40 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:35:40 xen-02 iscsi-ha: 13576 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:35:40 xen-02 iscsi-ha: 13576 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:35:40 xen-02 iscsi-ha: 13576 service_execute: systemctl mode being used
Dec 16 09:35:40 xen-02 iscsi-ha: 13576 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:35:40 xen-02 iscsi-ha: 13576 service_execute: Returning exit status [ 3 ]
Dec 16 09:35:40 xen-02 iscsi-ha: 13576 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:35:40 xen-02 iscsi-ha: 13576 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485528 dw:1087485528 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:35:40 xen-02 iscsi-ha: 13576 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:35:40 xen-02 iscsi-ha: 13576 DRBD Resource: iscsi1 in Connected state
Dec 16 09:35:40 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:40 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:42 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:45 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:35:45 xen-02 ha-lizard: 12028 Spawning new instance of ha-lizard
Dec 16 09:35:45 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:35:45 xen-02 ha-lizard:  This iteration is count 4459
Dec 16 09:35:45 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:45 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:35:46 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:35:46 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:35:46 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:35:46 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:35:46 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:35:47 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:35:47 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:35:47 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:35:47 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:35:47 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:35:47 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:35:47 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:35:47 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:35:47 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:35:47 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:35:47 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:35:47 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:35:47 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:35:47 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:35:47 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:35:47 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:35:47 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:35:47 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:35:47 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:48 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:35:48 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:35:48 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:35:48 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:35:48 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:35:48 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:35:49 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:35:49 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:35:49 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:35:49 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:35:50 xen-02 iscsi-ha: 13541 Spawning new instance of iscsi-ha
Dec 16 09:35:50 xen-02 iscsi-ha: 13541 check_logger_processes Checking logger processes
Dec 16 09:35:50 xen-02 iscsi-ha: 13541 check_logger_processes No processes to clear
Dec 16 09:35:50 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:35:50 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:35:50 xen-02 iscsi-ha:  This iteration is count 6651
Dec 16 09:35:50 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:50 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:35:50 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:35:50 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:35:50 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:35:50 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:35:50 xen-02 iscsi-ha: 14371 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:35:50 xen-02 iscsi-ha: 14371 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:35:50 xen-02 iscsi-ha: 14371 service_execute: systemctl mode being used
Dec 16 09:35:50 xen-02 iscsi-ha: 14371 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:35:50 xen-02 iscsi-ha: 14371 service_execute: Returning exit status [ 3 ]
Dec 16 09:35:50 xen-02 iscsi-ha: 14371 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:35:50 xen-02 iscsi-ha: 14371 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485528 dw:1087485528 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:35:50 xen-02 iscsi-ha: 14371 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:35:50 xen-02 iscsi-ha: 14371 DRBD Resource: iscsi1 in Connected state
Dec 16 09:35:50 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:50 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:52 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:55 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:35:57 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:00 xen-02 iscsi-ha: 14336 Spawning new instance of iscsi-ha
Dec 16 09:36:00 xen-02 iscsi-ha: 14336 check_logger_processes Checking logger processes
Dec 16 09:36:00 xen-02 iscsi-ha: 14336 check_logger_processes No processes to clear
Dec 16 09:36:00 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:36:00 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:36:00 xen-02 iscsi-ha:  This iteration is count 6652
Dec 16 09:36:00 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:00 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:36:00 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:36:00 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:36:00 xen-02 ha-lizard: 13674 Spawning new instance of ha-lizard
Dec 16 09:36:00 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:36:00 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:36:00 xen-02 iscsi-ha: 14549 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:36:00 xen-02 iscsi-ha: 14549 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:36:00 xen-02 iscsi-ha: 14549 service_execute: systemctl mode being used
Dec 16 09:36:01 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:36:01 xen-02 iscsi-ha: 14549 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:36:01 xen-02 ha-lizard:  This iteration is count 4460
Dec 16 09:36:01 xen-02 iscsi-ha: 14549 service_execute: Returning exit status [ 3 ]
Dec 16 09:36:01 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:01 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:36:01 xen-02 iscsi-ha: 14549 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:36:01 xen-02 iscsi-ha: 14549 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485532 dw:1087485532 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:36:01 xen-02 iscsi-ha: 14549 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:36:01 xen-02 iscsi-ha: 14549 DRBD Resource: iscsi1 in Connected state
Dec 16 09:36:01 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:01 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:01 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:36:01 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:36:01 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:36:01 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:36:01 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:36:02 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:36:02 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:36:02 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:36:02 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:36:02 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:36:02 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:36:02 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:36:02 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:36:02 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:36:02 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:36:02 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:36:02 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:36:02 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:36:02 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:36:02 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:36:02 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:36:02 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:36:02 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:36:02 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:03 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:36:03 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:36:03 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:36:03 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:36:03 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:36:03 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:36:03 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:36:04 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:36:04 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:36:04 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:36:05 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:36:07 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:10 xen-02 iscsi-ha: 14491 Spawning new instance of iscsi-ha
Dec 16 09:36:10 xen-02 iscsi-ha: 14491 check_logger_processes Checking logger processes
Dec 16 09:36:10 xen-02 iscsi-ha: 14491 check_logger_processes No processes to clear
Dec 16 09:36:10 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:36:10 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:36:10 xen-02 iscsi-ha:  This iteration is count 6653
Dec 16 09:36:10 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:10 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:36:10 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:36:10 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:36:11 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:36:11 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:36:11 xen-02 iscsi-ha: 15396 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:36:11 xen-02 iscsi-ha: 15396 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:36:11 xen-02 iscsi-ha: 15396 service_execute: systemctl mode being used
Dec 16 09:36:11 xen-02 iscsi-ha: 15396 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:36:11 xen-02 iscsi-ha: 15396 service_execute: Returning exit status [ 3 ]
Dec 16 09:36:11 xen-02 iscsi-ha: 15396 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:36:11 xen-02 iscsi-ha: 15396 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485532 dw:1087485532 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:36:11 xen-02 iscsi-ha: 15396 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:36:11 xen-02 iscsi-ha: 15396 DRBD Resource: iscsi1 in Connected state
Dec 16 09:36:11 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:11 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:12 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:15 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:36:16 xen-02 ha-lizard: 14533 Spawning new instance of ha-lizard
Dec 16 09:36:16 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:36:16 xen-02 ha-lizard:  This iteration is count 4461
Dec 16 09:36:16 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:16 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:36:16 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:36:16 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:36:16 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:36:16 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:36:16 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:36:17 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:36:17 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:36:17 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:36:17 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:36:17 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:36:17 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:36:17 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:36:17 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:36:17 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:36:17 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:36:17 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:36:17 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:36:17 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:36:17 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:36:17 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:36:17 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:36:17 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:36:17 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:18 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:36:18 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:36:18 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:36:18 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:36:18 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:36:18 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:36:18 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:36:19 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:36:19 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:36:19 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:36:19 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:36:21 xen-02 iscsi-ha: 15314 Spawning new instance of iscsi-ha
Dec 16 09:36:21 xen-02 iscsi-ha: 15314 check_logger_processes Checking logger processes
Dec 16 09:36:21 xen-02 iscsi-ha: 15314 check_logger_processes No processes to clear
Dec 16 09:36:21 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:36:21 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:36:21 xen-02 iscsi-ha:  This iteration is count 6654
Dec 16 09:36:21 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:21 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:36:21 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:36:21 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:36:21 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:36:21 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:36:21 xen-02 iscsi-ha: 16928 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:36:21 xen-02 iscsi-ha: 16928 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:36:21 xen-02 iscsi-ha: 16928 service_execute: systemctl mode being used
Dec 16 09:36:21 xen-02 iscsi-ha: 16928 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:36:21 xen-02 iscsi-ha: 16928 service_execute: Returning exit status [ 3 ]
Dec 16 09:36:21 xen-02 iscsi-ha: 16928 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:36:21 xen-02 iscsi-ha: 16928 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485532 dw:1087485532 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:36:21 xen-02 iscsi-ha: 16928 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:36:21 xen-02 iscsi-ha: 16928 DRBD Resource: iscsi1 in Connected state
Dec 16 09:36:21 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:21 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:22 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:25 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:36:27 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:31 xen-02 ha-lizard: 16241 Spawning new instance of ha-lizard
Dec 16 09:36:31 xen-02 iscsi-ha: 16893 Spawning new instance of iscsi-ha
Dec 16 09:36:31 xen-02 iscsi-ha: 16893 check_logger_processes Checking logger processes
Dec 16 09:36:31 xen-02 iscsi-ha: 16893 check_logger_processes No processes to clear
Dec 16 09:36:31 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:36:31 xen-02 ha-lizard:  This iteration is count 4462
Dec 16 09:36:31 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:31 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:36:31 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:36:31 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:36:31 xen-02 iscsi-ha:  This iteration is count 6655
Dec 16 09:36:31 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:31 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:36:31 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:36:31 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:36:31 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:36:31 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:36:31 xen-02 iscsi-ha: 17137 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:36:31 xen-02 iscsi-ha: 17137 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:36:31 xen-02 iscsi-ha: 17137 service_execute: systemctl mode being used
Dec 16 09:36:31 xen-02 iscsi-ha: 17137 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:36:31 xen-02 iscsi-ha: 17137 service_execute: Returning exit status [ 3 ]
Dec 16 09:36:31 xen-02 iscsi-ha: 17137 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:36:31 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:36:31 xen-02 iscsi-ha: 17137 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485540 dw:1087485540 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:36:31 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:36:31 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:36:31 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:36:31 xen-02 iscsi-ha: 17137 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:36:31 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:36:31 xen-02 iscsi-ha: 17137 DRBD Resource: iscsi1 in Connected state
Dec 16 09:36:31 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:31 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:32 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:36:32 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:36:32 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:36:32 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:36:32 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:36:32 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:36:32 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:36:32 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:36:32 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:36:32 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:36:32 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:36:32 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:36:32 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:36:33 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:33 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:36:33 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:36:33 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:36:33 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:36:33 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:36:33 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:36:33 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:36:33 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:36:33 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:36:33 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:36:34 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:36:34 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:36:34 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:36:34 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:36:34 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:36:35 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:36:38 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:41 xen-02 iscsi-ha: 17071 Spawning new instance of iscsi-ha
Dec 16 09:36:41 xen-02 iscsi-ha: 17071 check_logger_processes Checking logger processes
Dec 16 09:36:41 xen-02 iscsi-ha: 17071 check_logger_processes No processes to clear
Dec 16 09:36:41 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:36:41 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:36:41 xen-02 iscsi-ha:  This iteration is count 6656
Dec 16 09:36:41 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:41 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:36:41 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:36:41 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:36:41 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:36:41 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:36:41 xen-02 iscsi-ha: 17682 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:36:41 xen-02 iscsi-ha: 17682 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:36:41 xen-02 iscsi-ha: 17682 service_execute: systemctl mode being used
Dec 16 09:36:41 xen-02 iscsi-ha: 17682 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:36:41 xen-02 iscsi-ha: 17682 service_execute: Returning exit status [ 3 ]
Dec 16 09:36:41 xen-02 iscsi-ha: 17682 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:36:41 xen-02 iscsi-ha: 17682 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485540 dw:1087485540 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:36:41 xen-02 iscsi-ha: 17682 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:36:41 xen-02 iscsi-ha: 17682 DRBD Resource: iscsi1 in Connected state
Dec 16 09:36:41 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:41 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:43 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:45 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:36:46 xen-02 ha-lizard: 17054 Spawning new instance of ha-lizard
Dec 16 09:36:46 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:36:46 xen-02 ha-lizard:  This iteration is count 4463
Dec 16 09:36:46 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:46 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:36:46 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:36:46 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:36:46 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:36:46 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:36:46 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:36:47 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:36:47 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:36:47 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:36:47 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:36:47 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:36:47 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:36:47 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:36:47 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:36:47 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:36:47 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:36:48 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:48 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:36:48 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:36:48 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:36:48 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:36:48 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:36:48 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:36:48 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:36:48 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:36:48 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:36:48 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:36:49 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:36:49 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:36:49 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:36:49 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:36:49 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:36:49 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:36:49 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:36:49 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:36:51 xen-02 iscsi-ha: 17647 Spawning new instance of iscsi-ha
Dec 16 09:36:51 xen-02 iscsi-ha: 17647 check_logger_processes Checking logger processes
Dec 16 09:36:51 xen-02 iscsi-ha: 17647 check_logger_processes No processes to clear
Dec 16 09:36:51 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:36:51 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:36:51 xen-02 iscsi-ha:  This iteration is count 6657
Dec 16 09:36:51 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:51 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:36:51 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:36:51 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:36:51 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:36:51 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:36:51 xen-02 iscsi-ha: 19313 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:36:51 xen-02 iscsi-ha: 19313 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:36:51 xen-02 iscsi-ha: 19313 service_execute: systemctl mode being used
Dec 16 09:36:51 xen-02 iscsi-ha: 19313 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:36:51 xen-02 iscsi-ha: 19313 service_execute: Returning exit status [ 3 ]
Dec 16 09:36:51 xen-02 iscsi-ha: 19313 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:36:51 xen-02 iscsi-ha: 19313 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485540 dw:1087485540 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:36:51 xen-02 iscsi-ha: 19313 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:36:51 xen-02 iscsi-ha: 19313 DRBD Resource: iscsi1 in Connected state
Dec 16 09:36:51 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:51 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:53 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:55 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:36:58 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:01 xen-02 ha-lizard: 18085 Spawning new instance of ha-lizard
Dec 16 09:37:01 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:37:01 xen-02 ha-lizard:  This iteration is count 4464
Dec 16 09:37:01 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:01 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:37:01 xen-02 iscsi-ha: 19278 Spawning new instance of iscsi-ha
Dec 16 09:37:01 xen-02 iscsi-ha: 19278 check_logger_processes Checking logger processes
Dec 16 09:37:01 xen-02 iscsi-ha: 19278 check_logger_processes No processes to clear
Dec 16 09:37:01 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:37:01 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:37:01 xen-02 iscsi-ha:  This iteration is count 6658
Dec 16 09:37:01 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:01 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:37:01 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:37:01 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:37:01 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:37:01 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:37:01 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:37:01 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:37:01 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:37:01 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:37:01 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:37:01 xen-02 iscsi-ha: 19722 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:37:01 xen-02 iscsi-ha: 19722 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:37:01 xen-02 iscsi-ha: 19722 service_execute: systemctl mode being used
Dec 16 09:37:01 xen-02 iscsi-ha: 19722 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:37:01 xen-02 iscsi-ha: 19722 service_execute: Returning exit status [ 3 ]
Dec 16 09:37:01 xen-02 iscsi-ha: 19722 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:37:01 xen-02 iscsi-ha: 19722 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485544 dw:1087485544 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:37:02 xen-02 iscsi-ha: 19722 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:37:02 xen-02 iscsi-ha: 19722 DRBD Resource: iscsi1 in Connected state
Dec 16 09:37:02 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:02 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:02 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:37:02 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:37:02 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:37:02 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:37:02 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:37:02 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:37:02 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:37:03 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:37:03 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:37:03 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:37:03 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:03 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:37:03 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:37:03 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:37:03 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:37:03 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:37:03 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:37:03 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:37:03 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:37:03 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:37:04 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:37:04 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:37:04 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:37:04 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:37:04 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:37:04 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:37:04 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:37:04 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:37:04 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:37:05 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:37:08 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:11 xen-02 iscsi-ha: 19666 Spawning new instance of iscsi-ha
Dec 16 09:37:11 xen-02 iscsi-ha: 19666 check_logger_processes Checking logger processes
Dec 16 09:37:11 xen-02 iscsi-ha: 19666 check_logger_processes No processes to clear
Dec 16 09:37:11 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:37:11 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:37:11 xen-02 iscsi-ha:  This iteration is count 6659
Dec 16 09:37:11 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:11 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:37:11 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:37:11 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:37:12 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:37:12 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:37:12 xen-02 iscsi-ha: 20255 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:37:12 xen-02 iscsi-ha: 20255 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:37:12 xen-02 iscsi-ha: 20255 service_execute: systemctl mode being used
Dec 16 09:37:12 xen-02 iscsi-ha: 20255 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:37:12 xen-02 iscsi-ha: 20255 service_execute: Returning exit status [ 3 ]
Dec 16 09:37:12 xen-02 iscsi-ha: 20255 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:37:12 xen-02 iscsi-ha: 20255 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485544 dw:1087485544 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:37:12 xen-02 iscsi-ha: 20255 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:37:12 xen-02 iscsi-ha: 20255 DRBD Resource: iscsi1 in Connected state
Dec 16 09:37:12 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:12 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:13 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:15 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:37:16 xen-02 ha-lizard: 19608 Spawning new instance of ha-lizard
Dec 16 09:37:16 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:37:16 xen-02 ha-lizard:  This iteration is count 4465
Dec 16 09:37:16 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:16 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:37:16 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:37:16 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:37:16 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:37:16 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:37:16 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:37:17 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:37:17 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:37:17 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:37:18 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:37:18 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:37:18 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:37:18 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:37:18 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:18 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:37:18 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:37:18 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:37:18 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:37:18 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:37:18 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:37:18 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:37:18 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:37:18 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:37:18 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:37:18 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:37:18 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:37:18 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:37:19 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:37:19 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:37:19 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:37:19 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:37:19 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:37:19 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:37:19 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:37:19 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:37:21 xen-02 iscsi-ha: 20220 Spawning new instance of iscsi-ha
Dec 16 09:37:21 xen-02 iscsi-ha: 20220 check_logger_processes Checking logger processes
Dec 16 09:37:22 xen-02 iscsi-ha: 20220 check_logger_processes No processes to clear
Dec 16 09:37:22 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:37:22 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:37:22 xen-02 iscsi-ha:  This iteration is count 6660
Dec 16 09:37:22 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:22 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:37:22 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:37:22 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:37:22 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:37:22 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:37:22 xen-02 iscsi-ha: 21321 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:37:22 xen-02 iscsi-ha: 21321 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:37:22 xen-02 iscsi-ha: 21321 service_execute: systemctl mode being used
Dec 16 09:37:22 xen-02 iscsi-ha: 21321 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:37:22 xen-02 iscsi-ha: 21321 service_execute: Returning exit status [ 3 ]
Dec 16 09:37:22 xen-02 iscsi-ha: 21321 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:37:22 xen-02 iscsi-ha: 21321 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485544 dw:1087485544 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:37:22 xen-02 iscsi-ha: 21321 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:37:22 xen-02 iscsi-ha: 21321 DRBD Resource: iscsi1 in Connected state
Dec 16 09:37:22 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:22 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:23 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:25 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:37:28 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:31 xen-02 ha-lizard: 20348 Spawning new instance of ha-lizard
Dec 16 09:37:31 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:37:31 xen-02 ha-lizard:  This iteration is count 4466
Dec 16 09:37:31 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:31 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:37:32 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:37:32 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:37:32 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:37:32 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:37:32 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:37:32 xen-02 iscsi-ha: 21244 Spawning new instance of iscsi-ha
Dec 16 09:37:32 xen-02 iscsi-ha: 21244 check_logger_processes Checking logger processes
Dec 16 09:37:32 xen-02 iscsi-ha: 21244 check_logger_processes No processes to clear
Dec 16 09:37:32 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:37:32 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:37:32 xen-02 iscsi-ha:  This iteration is count 6661
Dec 16 09:37:32 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:32 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:37:32 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:37:32 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:37:32 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:37:32 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:37:32 xen-02 iscsi-ha: 22569 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:37:32 xen-02 iscsi-ha: 22569 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:37:32 xen-02 iscsi-ha: 22569 service_execute: systemctl mode being used
Dec 16 09:37:32 xen-02 iscsi-ha: 22569 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:37:32 xen-02 iscsi-ha: 22569 service_execute: Returning exit status [ 3 ]
Dec 16 09:37:32 xen-02 iscsi-ha: 22569 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:37:32 xen-02 iscsi-ha: 22569 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485552 dw:1087485552 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:37:32 xen-02 iscsi-ha: 22569 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:37:32 xen-02 iscsi-ha: 22569 DRBD Resource: iscsi1 in Connected state
Dec 16 09:37:32 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:32 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:33 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:37:33 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:37:33 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:37:33 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:33 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:37:33 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:37:33 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:37:33 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:37:33 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:37:33 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:37:33 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:37:33 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:37:33 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:37:33 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:37:33 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:37:33 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:37:33 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:37:33 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:37:33 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:37:33 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:37:34 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:37:34 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:37:34 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:37:34 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:37:34 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:37:34 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:37:34 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:37:34 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:37:34 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:37:35 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:37:38 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:42 xen-02 iscsi-ha: 22517 Spawning new instance of iscsi-ha
Dec 16 09:37:42 xen-02 iscsi-ha: 22517 check_logger_processes Checking logger processes
Dec 16 09:37:42 xen-02 iscsi-ha: 22517 check_logger_processes No processes to clear
Dec 16 09:37:42 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:37:42 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:37:42 xen-02 iscsi-ha:  This iteration is count 6662
Dec 16 09:37:42 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:42 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:37:42 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:37:42 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:37:42 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:37:42 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:37:42 xen-02 iscsi-ha: 23086 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:37:42 xen-02 iscsi-ha: 23086 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:37:42 xen-02 iscsi-ha: 23086 service_execute: systemctl mode being used
Dec 16 09:37:42 xen-02 iscsi-ha: 23086 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:37:42 xen-02 iscsi-ha: 23086 service_execute: Returning exit status [ 3 ]
Dec 16 09:37:42 xen-02 iscsi-ha: 23086 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:37:42 xen-02 iscsi-ha: 23086 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485552 dw:1087485552 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:37:42 xen-02 iscsi-ha: 23086 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:37:42 xen-02 iscsi-ha: 23086 DRBD Resource: iscsi1 in Connected state
Dec 16 09:37:42 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:42 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:43 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:45 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:37:46 xen-02 ha-lizard: 22438 Spawning new instance of ha-lizard
Dec 16 09:37:46 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:37:46 xen-02 ha-lizard:  This iteration is count 4467
Dec 16 09:37:46 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:46 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:37:47 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:37:47 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:37:47 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:37:47 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:37:47 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:37:48 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:48 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:37:48 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:37:48 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:37:48 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:37:48 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:37:48 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:37:48 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:37:48 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:37:48 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:37:48 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:37:48 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:37:48 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:37:48 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:37:48 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:37:48 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:37:48 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:37:48 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:37:48 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:37:49 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:37:49 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:37:49 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:37:49 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:37:49 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:37:49 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:37:49 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:37:49 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:37:49 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:37:49 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:37:52 xen-02 iscsi-ha: 23050 Spawning new instance of iscsi-ha
Dec 16 09:37:52 xen-02 iscsi-ha: 23050 check_logger_processes Checking logger processes
Dec 16 09:37:52 xen-02 iscsi-ha: 23050 check_logger_processes No processes to clear
Dec 16 09:37:52 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:37:52 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:37:52 xen-02 iscsi-ha:  This iteration is count 6663
Dec 16 09:37:52 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:52 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:37:52 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:37:52 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:37:52 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:37:52 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:37:52 xen-02 iscsi-ha: 23732 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:37:52 xen-02 iscsi-ha: 23732 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:37:52 xen-02 iscsi-ha: 23732 service_execute: systemctl mode being used
Dec 16 09:37:52 xen-02 iscsi-ha: 23732 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:37:52 xen-02 iscsi-ha: 23732 service_execute: Returning exit status [ 3 ]
Dec 16 09:37:52 xen-02 iscsi-ha: 23732 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:37:52 xen-02 iscsi-ha: 23732 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485552 dw:1087485552 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:37:52 xen-02 iscsi-ha: 23732 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:37:52 xen-02 iscsi-ha: 23732 DRBD Resource: iscsi1 in Connected state
Dec 16 09:37:52 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:52 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:53 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:55 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:37:58 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:01 xen-02 ha-lizard: 23175 Spawning new instance of ha-lizard
Dec 16 09:38:02 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:38:02 xen-02 ha-lizard:  This iteration is count 4468
Dec 16 09:38:02 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:02 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:38:02 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:38:02 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:38:02 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:38:02 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:38:02 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:38:02 xen-02 iscsi-ha: 23697 Spawning new instance of iscsi-ha
Dec 16 09:38:02 xen-02 iscsi-ha: 23697 check_logger_processes Checking logger processes
Dec 16 09:38:02 xen-02 iscsi-ha: 23697 check_logger_processes No processes to clear
Dec 16 09:38:02 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:38:02 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:38:02 xen-02 iscsi-ha:  This iteration is count 6664
Dec 16 09:38:02 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:02 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:38:02 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:38:02 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:38:02 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:38:02 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:38:02 xen-02 iscsi-ha: 25160 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:38:02 xen-02 iscsi-ha: 25160 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:38:02 xen-02 iscsi-ha: 25160 service_execute: systemctl mode being used
Dec 16 09:38:02 xen-02 iscsi-ha: 25160 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:38:02 xen-02 iscsi-ha: 25160 service_execute: Returning exit status [ 3 ]
Dec 16 09:38:02 xen-02 iscsi-ha: 25160 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:38:02 xen-02 iscsi-ha: 25160 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485556 dw:1087485556 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:38:02 xen-02 iscsi-ha: 25160 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:38:02 xen-02 iscsi-ha: 25160 DRBD Resource: iscsi1 in Connected state
Dec 16 09:38:03 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:03 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:03 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:03 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:38:03 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:38:03 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:38:03 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:38:03 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:38:03 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:38:03 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:38:03 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:38:03 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:38:03 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:38:03 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:38:03 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:38:03 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:38:03 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:38:03 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:38:03 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:38:03 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:38:03 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:38:04 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:38:04 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:38:04 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:38:04 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:38:04 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:38:04 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:38:04 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:38:05 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:38:05 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:38:05 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:38:05 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:38:08 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:12 xen-02 iscsi-ha: 25125 Spawning new instance of iscsi-ha
Dec 16 09:38:12 xen-02 iscsi-ha: 25125 check_logger_processes Checking logger processes
Dec 16 09:38:12 xen-02 iscsi-ha: 25125 check_logger_processes No processes to clear
Dec 16 09:38:12 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:38:12 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:38:12 xen-02 iscsi-ha:  This iteration is count 6665
Dec 16 09:38:12 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:12 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:38:12 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:38:12 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:38:13 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:38:13 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:38:13 xen-02 iscsi-ha: 25693 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:38:13 xen-02 iscsi-ha: 25693 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:38:13 xen-02 iscsi-ha: 25693 service_execute: systemctl mode being used
Dec 16 09:38:13 xen-02 iscsi-ha: 25693 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:38:13 xen-02 iscsi-ha: 25693 service_execute: Returning exit status [ 3 ]
Dec 16 09:38:13 xen-02 iscsi-ha: 25693 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:38:13 xen-02 iscsi-ha: 25693 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485556 dw:1087485556 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:38:13 xen-02 iscsi-ha: 25693 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:38:13 xen-02 iscsi-ha: 25693 DRBD Resource: iscsi1 in Connected state
Dec 16 09:38:13 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:13 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:13 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:15 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:38:17 xen-02 ha-lizard: 25034 Spawning new instance of ha-lizard
Dec 16 09:38:17 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:38:17 xen-02 ha-lizard:  This iteration is count 4469
Dec 16 09:38:17 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:17 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:38:17 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:38:17 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:38:17 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:38:17 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:38:17 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:38:18 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:18 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:38:18 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:38:18 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:38:18 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:38:18 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:38:18 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:38:18 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:38:18 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:38:18 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:38:18 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:38:18 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:38:18 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:38:18 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:38:19 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:38:19 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:38:19 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:38:19 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:38:19 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:38:19 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:38:19 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:38:19 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:38:19 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:38:19 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:38:20 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:38:20 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:38:20 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:38:20 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:38:20 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:38:22 xen-02 iscsi-ha: 25658 Spawning new instance of iscsi-ha
Dec 16 09:38:22 xen-02 iscsi-ha: 25658 check_logger_processes Checking logger processes
Dec 16 09:38:22 xen-02 iscsi-ha: 25658 check_logger_processes No processes to clear
Dec 16 09:38:23 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:38:23 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:38:23 xen-02 iscsi-ha:  This iteration is count 6666
Dec 16 09:38:23 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:23 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:38:23 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:38:23 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:38:23 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:38:23 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:38:23 xen-02 iscsi-ha: 26294 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:38:23 xen-02 iscsi-ha: 26294 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:38:23 xen-02 iscsi-ha: 26294 service_execute: systemctl mode being used
Dec 16 09:38:23 xen-02 iscsi-ha: 26294 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:38:23 xen-02 iscsi-ha: 26294 service_execute: Returning exit status [ 3 ]
Dec 16 09:38:23 xen-02 iscsi-ha: 26294 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:38:23 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:23 xen-02 iscsi-ha: 26294 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485556 dw:1087485556 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:38:23 xen-02 iscsi-ha: 26294 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:38:23 xen-02 iscsi-ha: 26294 DRBD Resource: iscsi1 in Connected state
Dec 16 09:38:23 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:23 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:25 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:38:28 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:32 xen-02 ha-lizard: 25774 Spawning new instance of ha-lizard
Dec 16 09:38:32 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:38:32 xen-02 ha-lizard:  This iteration is count 4470
Dec 16 09:38:32 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:32 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:38:32 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:38:32 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:38:32 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:38:32 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:38:32 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:38:33 xen-02 iscsi-ha: 26259 Spawning new instance of iscsi-ha
Dec 16 09:38:33 xen-02 iscsi-ha: 26259 check_logger_processes Checking logger processes
Dec 16 09:38:33 xen-02 iscsi-ha: 26259 check_logger_processes No processes to clear
Dec 16 09:38:33 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:38:33 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:38:33 xen-02 iscsi-ha:  This iteration is count 6667
Dec 16 09:38:33 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:33 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:38:33 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:38:33 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:38:33 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:33 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:38:33 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:38:33 xen-02 iscsi-ha: 27367 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:38:33 xen-02 iscsi-ha: 27367 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:38:33 xen-02 iscsi-ha: 27367 service_execute: systemctl mode being used
Dec 16 09:38:33 xen-02 iscsi-ha: 27367 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:38:33 xen-02 iscsi-ha: 27367 service_execute: Returning exit status [ 3 ]
Dec 16 09:38:33 xen-02 iscsi-ha: 27367 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:38:33 xen-02 iscsi-ha: 27367 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485564 dw:1087485564 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:38:33 xen-02 iscsi-ha: 27367 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:38:33 xen-02 iscsi-ha: 27367 DRBD Resource: iscsi1 in Connected state
Dec 16 09:38:33 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:33 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:33 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:38:33 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:38:33 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:38:33 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:38:33 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:38:33 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:38:33 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:38:33 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:38:33 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:38:33 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:38:33 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:38:33 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:38:33 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:38:34 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:38:34 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:38:34 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:38:34 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:38:34 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:38:34 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:38:34 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:38:34 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:38:34 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:38:34 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:38:35 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:38:35 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:38:35 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:38:35 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:38:35 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:38:35 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:38:38 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:43 xen-02 iscsi-ha: 27291 Spawning new instance of iscsi-ha
Dec 16 09:38:43 xen-02 iscsi-ha: 27291 check_logger_processes Checking logger processes
Dec 16 09:38:43 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:43 xen-02 iscsi-ha: 27291 check_logger_processes No processes to clear
Dec 16 09:38:43 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:38:43 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:38:43 xen-02 iscsi-ha:  This iteration is count 6668
Dec 16 09:38:43 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:43 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:38:43 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:38:43 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:38:43 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:38:43 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:38:43 xen-02 iscsi-ha: 28249 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:38:43 xen-02 iscsi-ha: 28249 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:38:43 xen-02 iscsi-ha: 28249 service_execute: systemctl mode being used
Dec 16 09:38:43 xen-02 iscsi-ha: 28249 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:38:43 xen-02 iscsi-ha: 28249 service_execute: Returning exit status [ 3 ]
Dec 16 09:38:43 xen-02 iscsi-ha: 28249 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:38:43 xen-02 iscsi-ha: 28249 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485564 dw:1087485564 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:38:43 xen-02 iscsi-ha: 28249 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:38:43 xen-02 iscsi-ha: 28249 DRBD Resource: iscsi1 in Connected state
Dec 16 09:38:43 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:43 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:45 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:38:47 xen-02 ha-lizard: 27015 Spawning new instance of ha-lizard
Dec 16 09:38:47 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:38:47 xen-02 ha-lizard:  This iteration is count 4471
Dec 16 09:38:47 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:47 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:38:47 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:38:47 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:38:47 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:38:47 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:38:47 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:38:48 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:48 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:38:48 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:38:48 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:38:48 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:38:48 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:38:48 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:38:48 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:38:48 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:38:48 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:38:48 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:38:49 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:38:49 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:38:49 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:38:49 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:38:49 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:38:49 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:38:49 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:38:49 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:38:49 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:38:49 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:38:49 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:38:50 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:38:50 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:38:50 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:38:50 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:38:50 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:38:50 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:38:50 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:38:53 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:53 xen-02 iscsi-ha: 28212 Spawning new instance of iscsi-ha
Dec 16 09:38:53 xen-02 iscsi-ha: 28212 check_logger_processes Checking logger processes
Dec 16 09:38:53 xen-02 iscsi-ha: 28212 check_logger_processes No processes to clear
Dec 16 09:38:53 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:38:53 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:38:53 xen-02 iscsi-ha:  This iteration is count 6669
Dec 16 09:38:53 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:53 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:38:53 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:38:53 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:38:53 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:38:53 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:38:53 xen-02 iscsi-ha: 28863 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:38:53 xen-02 iscsi-ha: 28863 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:38:53 xen-02 iscsi-ha: 28863 service_execute: systemctl mode being used
Dec 16 09:38:53 xen-02 iscsi-ha: 28863 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:38:53 xen-02 iscsi-ha: 28863 service_execute: Returning exit status [ 3 ]
Dec 16 09:38:53 xen-02 iscsi-ha: 28863 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:38:53 xen-02 iscsi-ha: 28863 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485564 dw:1087485564 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:38:53 xen-02 iscsi-ha: 28863 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:38:53 xen-02 iscsi-ha: 28863 DRBD Resource: iscsi1 in Connected state
Dec 16 09:38:53 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:53 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:55 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:38:58 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:02 xen-02 ha-lizard: 28334 Spawning new instance of ha-lizard
Dec 16 09:39:02 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:39:02 xen-02 ha-lizard:  This iteration is count 4472
Dec 16 09:39:02 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:02 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:39:02 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:39:02 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:39:02 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:39:02 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:39:02 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:39:03 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:03 xen-02 iscsi-ha: 28828 Spawning new instance of iscsi-ha
Dec 16 09:39:03 xen-02 iscsi-ha: 28828 check_logger_processes Checking logger processes
Dec 16 09:39:03 xen-02 iscsi-ha: 28828 check_logger_processes No processes to clear
Dec 16 09:39:03 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:39:03 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:39:03 xen-02 iscsi-ha:  This iteration is count 6670
Dec 16 09:39:03 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:03 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:39:03 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:39:03 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:39:03 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:39:03 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:39:03 xen-02 iscsi-ha: 29270 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:39:03 xen-02 iscsi-ha: 29270 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:39:03 xen-02 iscsi-ha: 29270 service_execute: systemctl mode being used
Dec 16 09:39:03 xen-02 iscsi-ha: 29270 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:39:03 xen-02 iscsi-ha: 29270 service_execute: Returning exit status [ 3 ]
Dec 16 09:39:03 xen-02 iscsi-ha: 29270 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:39:03 xen-02 iscsi-ha: 29270 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485568 dw:1087485568 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:39:03 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:39:03 xen-02 iscsi-ha: 29270 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:39:03 xen-02 iscsi-ha: 29270 DRBD Resource: iscsi1 in Connected state
Dec 16 09:39:03 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:39:03 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:39:03 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:39:03 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:03 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:04 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:39:04 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:39:04 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:39:04 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:39:04 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:39:04 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:39:04 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:39:04 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:39:04 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:39:04 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:39:04 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:39:04 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:39:04 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:39:04 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:39:04 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:39:04 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:39:05 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:39:05 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:39:05 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:39:05 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:39:05 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:39:05 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:39:05 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:39:05 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:39:05 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:39:08 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:13 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:13 xen-02 iscsi-ha: 29235 Spawning new instance of iscsi-ha
Dec 16 09:39:13 xen-02 iscsi-ha: 29235 check_logger_processes Checking logger processes
Dec 16 09:39:13 xen-02 iscsi-ha: 29235 check_logger_processes No processes to clear
Dec 16 09:39:13 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:39:13 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:39:13 xen-02 iscsi-ha:  This iteration is count 6671
Dec 16 09:39:13 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:13 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:39:13 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:39:13 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:39:13 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:39:13 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:39:13 xen-02 iscsi-ha: 30811 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:39:13 xen-02 iscsi-ha: 30811 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:39:13 xen-02 iscsi-ha: 30811 service_execute: systemctl mode being used
Dec 16 09:39:14 xen-02 iscsi-ha: 30811 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:39:14 xen-02 iscsi-ha: 30811 service_execute: Returning exit status [ 3 ]
Dec 16 09:39:14 xen-02 iscsi-ha: 30811 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:39:14 xen-02 iscsi-ha: 30811 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485568 dw:1087485568 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:39:14 xen-02 iscsi-ha: 30811 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:39:14 xen-02 iscsi-ha: 30811 DRBD Resource: iscsi1 in Connected state
Dec 16 09:39:14 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:14 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:15 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:39:17 xen-02 ha-lizard: 29142 Spawning new instance of ha-lizard
Dec 16 09:39:17 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:39:17 xen-02 ha-lizard:  This iteration is count 4473
Dec 16 09:39:17 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:17 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:39:17 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:39:17 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:39:17 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:39:17 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:39:17 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:39:18 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:19 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:39:19 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:39:19 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:39:19 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:39:19 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:39:19 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:39:19 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:39:19 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:39:19 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:39:19 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:39:19 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:39:19 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:39:19 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:39:19 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:39:19 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:39:19 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:39:19 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:39:19 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:39:19 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:39:20 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:39:20 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:39:20 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:39:20 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:39:20 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:39:20 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:39:20 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:39:20 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:39:20 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:39:23 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:23 xen-02 iscsi-ha: 30776 Spawning new instance of iscsi-ha
Dec 16 09:39:23 xen-02 iscsi-ha: 30776 check_logger_processes Checking logger processes
Dec 16 09:39:23 xen-02 iscsi-ha: 30776 check_logger_processes No processes to clear
Dec 16 09:39:23 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:39:23 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:39:23 xen-02 iscsi-ha:  This iteration is count 6672
Dec 16 09:39:23 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:23 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:39:23 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:39:23 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:39:24 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:39:24 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:39:24 xen-02 iscsi-ha: 31410 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:39:24 xen-02 iscsi-ha: 31410 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:39:24 xen-02 iscsi-ha: 31410 service_execute: systemctl mode being used
Dec 16 09:39:24 xen-02 iscsi-ha: 31410 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:39:24 xen-02 iscsi-ha: 31410 service_execute: Returning exit status [ 3 ]
Dec 16 09:39:24 xen-02 iscsi-ha: 31410 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:39:24 xen-02 iscsi-ha: 31410 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485568 dw:1087485568 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:39:24 xen-02 iscsi-ha: 31410 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:39:24 xen-02 iscsi-ha: 31410 DRBD Resource: iscsi1 in Connected state
Dec 16 09:39:24 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:24 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:25 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:39:28 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:32 xen-02 ha-lizard: 30890 Spawning new instance of ha-lizard
Dec 16 09:39:32 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:39:32 xen-02 ha-lizard:  This iteration is count 4474
Dec 16 09:39:32 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:32 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:39:33 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:39:33 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:39:33 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:39:33 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:39:33 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:39:33 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:34 xen-02 iscsi-ha: 31375 Spawning new instance of iscsi-ha
Dec 16 09:39:34 xen-02 iscsi-ha: 31375 check_logger_processes Checking logger processes
Dec 16 09:39:34 xen-02 iscsi-ha: 31375 check_logger_processes No processes to clear
Dec 16 09:39:34 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:39:34 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:39:34 xen-02 iscsi-ha:  This iteration is count 6673
Dec 16 09:39:34 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:34 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:39:34 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:39:34 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:39:34 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:39:34 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:39:34 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:39:34 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:39:34 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:39:34 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:39:34 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:39:34 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:39:34 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:39:34 xen-02 iscsi-ha: 31841 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:39:34 xen-02 iscsi-ha: 31841 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:39:34 xen-02 iscsi-ha: 31841 service_execute: systemctl mode being used
Dec 16 09:39:34 xen-02 iscsi-ha: 31841 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:39:34 xen-02 iscsi-ha: 31841 service_execute: Returning exit status [ 3 ]
Dec 16 09:39:34 xen-02 iscsi-ha: 31841 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:39:34 xen-02 iscsi-ha: 31841 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485576 dw:1087485576 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:39:34 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:39:34 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:39:34 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:39:34 xen-02 iscsi-ha: 31841 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:39:34 xen-02 iscsi-ha: 31841 DRBD Resource: iscsi1 in Connected state
Dec 16 09:39:34 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:34 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:34 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:39:34 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:39:34 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:39:34 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:39:34 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:39:34 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:39:34 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:39:34 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:39:35 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:39:35 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:39:35 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:39:35 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:39:35 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:39:35 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:39:35 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:39:35 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:39:35 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:39:35 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:39:35 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:39:38 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:43 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:44 xen-02 iscsi-ha: 31784 Spawning new instance of iscsi-ha
Dec 16 09:39:44 xen-02 iscsi-ha: 31784 check_logger_processes Checking logger processes
Dec 16 09:39:44 xen-02 iscsi-ha: 31784 check_logger_processes No processes to clear
Dec 16 09:39:44 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:39:44 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:39:44 xen-02 iscsi-ha:  This iteration is count 6674
Dec 16 09:39:44 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:44 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:39:44 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:39:44 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:39:44 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:39:44 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:39:44 xen-02 iscsi-ha: 758 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:39:44 xen-02 iscsi-ha: 758 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:39:44 xen-02 iscsi-ha: 758 service_execute: systemctl mode being used
Dec 16 09:39:44 xen-02 iscsi-ha: 758 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:39:44 xen-02 iscsi-ha: 758 service_execute: Returning exit status [ 3 ]
Dec 16 09:39:44 xen-02 iscsi-ha: 758 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:39:44 xen-02 iscsi-ha: 758 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485576 dw:1087485576 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:39:44 xen-02 iscsi-ha: 758 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:39:44 xen-02 iscsi-ha: 758 DRBD Resource: iscsi1 in Connected state
Dec 16 09:39:44 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:44 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:45 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:39:47 xen-02 ha-lizard: 31702 Spawning new instance of ha-lizard
Dec 16 09:39:47 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:39:47 xen-02 ha-lizard:  This iteration is count 4475
Dec 16 09:39:47 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:47 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:39:48 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:39:48 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:39:48 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:39:48 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:39:48 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:39:48 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:49 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:39:49 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:39:49 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:39:49 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:39:49 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:39:49 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:39:49 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:39:49 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:39:49 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:39:49 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:39:49 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:39:49 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:39:49 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:39:49 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:39:49 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:39:49 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:39:49 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:39:49 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:39:50 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:39:50 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:39:50 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:39:50 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:39:50 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:39:50 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:39:50 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:39:50 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:39:51 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:39:51 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:39:53 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:54 xen-02 iscsi-ha: 682 Spawning new instance of iscsi-ha
Dec 16 09:39:54 xen-02 iscsi-ha: 682 check_logger_processes Checking logger processes
Dec 16 09:39:54 xen-02 iscsi-ha: 682 check_logger_processes No processes to clear
Dec 16 09:39:54 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:39:54 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:39:54 xen-02 iscsi-ha:  This iteration is count 6675
Dec 16 09:39:54 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:54 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:39:54 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:39:54 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:39:54 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:39:54 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:39:54 xen-02 iscsi-ha: 1749 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:39:54 xen-02 iscsi-ha: 1749 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:39:54 xen-02 iscsi-ha: 1749 service_execute: systemctl mode being used
Dec 16 09:39:54 xen-02 iscsi-ha: 1749 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:39:54 xen-02 iscsi-ha: 1749 service_execute: Returning exit status [ 3 ]
Dec 16 09:39:54 xen-02 iscsi-ha: 1749 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:39:54 xen-02 iscsi-ha: 1749 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485576 dw:1087485576 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:39:54 xen-02 iscsi-ha: 1749 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:39:54 xen-02 iscsi-ha: 1749 DRBD Resource: iscsi1 in Connected state
Dec 16 09:39:54 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:54 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:55 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:39:58 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:02 xen-02 ha-lizard: 1020 Spawning new instance of ha-lizard
Dec 16 09:40:03 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:03 xen-02 ha-lizard:  This iteration is count 4476
Dec 16 09:40:03 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:03 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:40:03 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:04 xen-02 iscsi-ha: 1714 Spawning new instance of iscsi-ha
Dec 16 09:40:04 xen-02 iscsi-ha: 1714 check_logger_processes Checking logger processes
Dec 16 09:40:04 xen-02 iscsi-ha: 1714 check_logger_processes No processes to clear
Dec 16 09:40:04 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:40:04 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:04 xen-02 iscsi-ha:  This iteration is count 6676
Dec 16 09:40:04 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:04 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:40:04 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:40:04 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:40:04 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:40:04 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:40:04 xen-02 iscsi-ha: 2261 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:40:04 xen-02 iscsi-ha: 2261 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:40:04 xen-02 iscsi-ha: 2261 service_execute: systemctl mode being used
Dec 16 09:40:04 xen-02 iscsi-ha: 2261 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:40:04 xen-02 iscsi-ha: 2261 service_execute: Returning exit status [ 3 ]
Dec 16 09:40:04 xen-02 iscsi-ha: 2261 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:40:04 xen-02 iscsi-ha: 2261 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485580 dw:1087485580 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:40:04 xen-02 iscsi-ha: 2261 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:40:04 xen-02 iscsi-ha: 2261 DRBD Resource: iscsi1 in Connected state
Dec 16 09:40:05 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:40:08 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:13 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:14 xen-02 iscsi-ha: 2224 Spawning new instance of iscsi-ha
Dec 16 09:40:14 xen-02 iscsi-ha: 2224 check_logger_processes Checking logger processes
Dec 16 09:40:14 xen-02 iscsi-ha: 2224 check_logger_processes No processes to clear
Dec 16 09:40:14 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:40:14 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:14 xen-02 iscsi-ha:  This iteration is count 6677
Dec 16 09:40:14 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:14 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:40:14 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:40:14 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:40:14 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:40:14 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:40:14 xen-02 iscsi-ha: 2423 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:40:14 xen-02 iscsi-ha: 2423 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:40:14 xen-02 iscsi-ha: 2423 service_execute: systemctl mode being used
Dec 16 09:40:14 xen-02 iscsi-ha: 2423 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:40:14 xen-02 iscsi-ha: 2423 service_execute: Returning exit status [ 3 ]
Dec 16 09:40:14 xen-02 iscsi-ha: 2423 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:40:14 xen-02 iscsi-ha: 2423 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485580 dw:1087485580 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:40:14 xen-02 iscsi-ha: 2423 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:40:15 xen-02 iscsi-ha: 2423 DRBD Resource: iscsi1 in Connected state
Dec 16 09:40:15 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:40:18 xen-02 ha-lizard: 2148 ha-lizard already running: Attempt 1 on PIDS: 2148
Dec 16 09:40:18 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:22 xen-02 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: Unhandled exception
Dec 16 09:40:22 xen-02 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: Sys_error("Broken pipe")
Dec 16 09:40:22 xen-02 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: Fatal error: exception Sys_error("Broken pipe")
Dec 16 09:40:22 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:40:22 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:40:22 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:40:22 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:40:22 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:40:22 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:22 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:23 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:23 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:40:23 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:40:23 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:40:23 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:40:23 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:40:23 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:40:23 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:40:23 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:40:23 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:40:23 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:40:24 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:24 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:40:24 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:40:24 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:40:24 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:40:24 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:40:24 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:40:24 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:40:24 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:24 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:40:24 xen-02 iscsi-ha: 2388 Spawning new instance of iscsi-ha
Dec 16 09:40:24 xen-02 iscsi-ha: 2388 check_logger_processes Checking logger processes
Dec 16 09:40:24 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:24 xen-02 ha-lizard:  check_email_enabled: Email enabled for write_pool_state
Dec 16 09:40:24 xen-02 ha-lizard:  email: Sending ALERT email to root@localhost: write_pool_state:  Error retrieving autopromote_uuid from pool configuration
Dec 16 09:40:24 xen-02 iscsi-ha: 2388 check_logger_processes No processes to clear
Dec 16 09:40:24 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:40:24 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:24 xen-02 iscsi-ha:  This iteration is count 6678
Dec 16 09:40:24 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:24 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:40:24 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:40:24 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:40:24 xen-02 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Traceback (most recent call last):
Dec 16 09:40:24 xen-02 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/etc/ha-lizard/scripts/email_alert.py", line 105, in <module>
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email from: root@localhost
Dec 16 09:40:24 xen-02 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     message = smtplib.SMTP(smtp_server, smtp_port, hostname)
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email to: root@localhost
Dec 16 09:40:24 xen-02 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xen-02
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Timestamp: Fri Dec 16 09:40:24 GMT 2016
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Process: ha-lizard Version: 2.1.3
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Content: write_pool_state: Error retrieving autopromote_uuid from pool configuration
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: 
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool UUID       : e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Name-Label : XenPool
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master UUID: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:24 xen-02 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     (code, msg) = self.connect(host, port)
Dec 16 09:40:24 xen-02 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 315, in connect
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-01
Dec 16 09:40:24 xen-02 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     self.sock = self._get_socket(host, port, self.timeout)
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : true
Dec 16 09:40:24 xen-02 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket
Dec 16 09:40:24 xen-02 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     return socket.create_connection((host, port), timeout)
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-02
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:24 xen-02 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : false
Dec 16 09:40:24 xen-02 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Hostname: xen-02
Dec 16 09:40:24 xen-02 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     raise err
Dec 16 09:40:24 xen-02 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: socket.error: [Errno 111] Connection refused
Dec 16 09:40:24 xen-02 ha-lizard:  email: Message copied to /dev/shm/ha-lizard-mail/1481881224.msg - Suppressing duplicates for 60 Minutes
Dec 16 09:40:25 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:40:25 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:40:25 xen-02 iscsi-ha: 3988 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:40:25 xen-02 iscsi-ha: 3988 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:40:25 xen-02 iscsi-ha: 3988 service_execute: systemctl mode being used
Dec 16 09:40:25 xen-02 iscsi-ha: 3988 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:40:25 xen-02 iscsi-ha: 3988 service_execute: Returning exit status [ 3 ]
Dec 16 09:40:25 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:25 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:40:25 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:40:25 xen-02 iscsi-ha: 3988 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:40:25 xen-02 iscsi-ha: 3988 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485580 dw:1087485580 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:40:25 xen-02 iscsi-ha: 3988 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:40:25 xen-02 iscsi-ha: 3988 DRBD Resource: iscsi1 in Connected state
Dec 16 09:40:25 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:25 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:25 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:40:25 xen-02 ha-lizard:  get_pool_host_list: returned bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:25 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:40:25 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.200
Dec 16 09:40:25 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:40:28 xen-02 ha-lizard: 2148 Spawning new instance of ha-lizard
Dec 16 09:40:28 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:28 xen-02 ha-lizard:  This iteration is count 4477
Dec 16 09:40:28 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:28 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:40:28 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:28 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:40:28 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:40:28 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:40:28 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:40:28 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:40:29 xen-02 ha-lizard:  This slave- xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce not permitted to become master
Dec 16 09:40:29 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:40:29 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:40:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:40:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:40:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:40:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:40:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:40:29 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:40:29 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:40:29 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:29 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:40:29 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:40:30 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:40:30 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:40:30 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:40:30 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:40:30 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:40:30 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:30 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:40:30 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:30 xen-02 ha-lizard:  check_email_enabled: Email enabled for write_pool_state
Dec 16 09:40:30 xen-02 ha-lizard:  email: Duplicate message - not sending. Content = write_pool_state:  Error retrieving autopromote_uuid from pool configuration
Dec 16 09:40:30 xen-02 ha-lizard:  email: Message barred for 60 minutes
Dec 16 09:40:30 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:30 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:40:30 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:40:31 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:40:31 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:31 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:40:31 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:40:31 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:40:33 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:34 xen-02 iscsi-ha: 3923 Spawning new instance of iscsi-ha
Dec 16 09:40:34 xen-02 iscsi-ha: 3923 check_logger_processes Checking logger processes
Dec 16 09:40:35 xen-02 iscsi-ha: 3923 check_logger_processes No processes to clear
Dec 16 09:40:35 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:40:35 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:35 xen-02 iscsi-ha:  This iteration is count 6679
Dec 16 09:40:35 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:35 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:40:35 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:40:35 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:40:35 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:40:35 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:40:35 xen-02 iscsi-ha: 5030 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:40:35 xen-02 iscsi-ha: 5030 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:40:35 xen-02 iscsi-ha: 5030 service_execute: systemctl mode being used
Dec 16 09:40:35 xen-02 iscsi-ha: 5030 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:40:35 xen-02 iscsi-ha: 5030 service_execute: Returning exit status [ 3 ]
Dec 16 09:40:35 xen-02 iscsi-ha: 5030 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:40:35 xen-02 iscsi-ha: 5030 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485588 dw:1087485588 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:40:35 xen-02 iscsi-ha: 5030 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:40:35 xen-02 iscsi-ha: 5030 DRBD Resource: iscsi1 in Connected state
Dec 16 09:40:35 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:35 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:35 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:40:38 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:43 xen-02 ha-lizard: 4285 Spawning new instance of ha-lizard
Dec 16 09:40:43 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:43 xen-02 ha-lizard:  This iteration is count 4478
Dec 16 09:40:43 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:43 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:40:43 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:43 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:40:43 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:40:43 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:40:43 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:40:43 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:40:44 xen-02 ha-lizard:  This slave- xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce not permitted to become master
Dec 16 09:40:44 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:40:44 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:40:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:40:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:40:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:40:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:40:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:40:44 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:40:44 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:40:45 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:45 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:40:45 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:40:45 xen-02 iscsi-ha: 4995 Spawning new instance of iscsi-ha
Dec 16 09:40:45 xen-02 iscsi-ha: 4995 check_logger_processes Checking logger processes
Dec 16 09:40:45 xen-02 iscsi-ha: 4995 check_logger_processes No processes to clear
Dec 16 09:40:45 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:40:45 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:45 xen-02 iscsi-ha:  This iteration is count 6680
Dec 16 09:40:45 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:40:45 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:45 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:40:45 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:40:45 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:40:45 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:40:45 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:40:45 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:40:45 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:40:45 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:40:45 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:40:45 xen-02 iscsi-ha: 5376 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:40:45 xen-02 iscsi-ha: 5376 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:40:45 xen-02 iscsi-ha: 5376 service_execute: systemctl mode being used
Dec 16 09:40:45 xen-02 iscsi-ha: 5376 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:40:45 xen-02 iscsi-ha: 5376 service_execute: Returning exit status [ 3 ]
Dec 16 09:40:45 xen-02 iscsi-ha: 5376 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:40:45 xen-02 iscsi-ha: 5376 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485588 dw:1087485588 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:40:45 xen-02 iscsi-ha: 5376 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:40:45 xen-02 iscsi-ha: 5376 DRBD Resource: iscsi1 in Connected state
Dec 16 09:40:45 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:40:45 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:45 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:45 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:45 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:40:46 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:46 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:40:46 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:40:46 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:40:46 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:46 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:40:46 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:40:46 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:40:48 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:53 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:55 xen-02 iscsi-ha: 5307 Spawning new instance of iscsi-ha
Dec 16 09:40:55 xen-02 iscsi-ha: 5307 check_logger_processes Checking logger processes
Dec 16 09:40:55 xen-02 iscsi-ha: 5307 check_logger_processes No processes to clear
Dec 16 09:40:55 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:40:55 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:55 xen-02 iscsi-ha:  This iteration is count 6681
Dec 16 09:40:55 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:55 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:40:55 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:40:55 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:40:55 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:40:55 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:40:55 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:40:55 xen-02 iscsi-ha: 7010 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:40:55 xen-02 iscsi-ha: 7010 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:40:55 xen-02 iscsi-ha: 7010 service_execute: systemctl mode being used
Dec 16 09:40:55 xen-02 iscsi-ha: 7010 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:40:55 xen-02 iscsi-ha: 7010 service_execute: Returning exit status [ 3 ]
Dec 16 09:40:55 xen-02 iscsi-ha: 7010 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:40:55 xen-02 iscsi-ha: 7010 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485588 dw:1087485588 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:40:55 xen-02 iscsi-ha: 7010 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:40:55 xen-02 iscsi-ha: 7010 DRBD Resource: iscsi1 in Connected state
Dec 16 09:40:55 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:55 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:57 xen-02 tapback: tapback.c:583 Identified blktap2 device major = 254
Dec 16 09:40:57 xen-02 tapback: tapback.c:637 only serving domain 7
Dec 16 09:40:57 xen-02 scripts-block: backend/vbd3/7/768: add params="/dev/sm/backend/b88adcc4-b907-8315-6aa3-b9d781c78c5e/3d66fc17-d503-4bf4-aad8-7a4619a8425a"
Dec 16 09:40:57 xen-02 scripts-block: backend/vbd3/7/768: physical-device=fe:1
Dec 16 09:40:57 xen-02 scripts-block: backend/vbd3/7/768: writing physical-device=device
Dec 16 09:40:57 xen-02 scripts-block: backend/vbd3/7/5696: add params=""
Dec 16 09:40:58 xen-02 scripts-block: backend/vbd3/7/5696: not writing physical-device because no-physical-device is present
Dec 16 09:40:58 xen-02 vif-real: Called as: /usr/libexec/xenopsd/vif-real vif 7 0 add
Dec 16 09:40:58 xen-02 scripts-vif: Called as "vif 7 0 add" domid:7 devid:0 mode:openvswitch
Dec 16 09:40:58 xen-02 vif-real: Called as: /usr/libexec/xenopsd/vif-real vif 7 0 online
Dec 16 09:40:58 xen-02 scripts-vif: Called as "vif 7 0 online" domid:7 devid:0 mode:openvswitch
Dec 16 09:40:58 xen-02 scripts-vif: Setting vif7.0 MTU 1500
Dec 16 09:40:58 xen-02 scripts-vif: Failed to read backend/vif/7/0/bridge from xenstore. Trying /xapi/174112ca-28d4-bbfe-6036-a2ee7a34e444/private/vif/0/bridge
Dec 16 09:40:58 xen-02 scripts-vif: Adding vif7.0 to xenbr5 with address fe:ff:ff:ff:ff:ff
Dec 16 09:40:58 xen-02 scripts-vif: vif7.0 external-ids:"xs-vm-uuid"="174112ca-28d4-bbfe-6036-a2ee7a34e444"
Dec 16 09:40:58 xen-02 scripts-vif: vif7.0 external-ids:"xs-vif-uuid"="48689027-c5fa-ba0e-4022-99bca196431b"
Dec 16 09:40:58 xen-02 scripts-vif: vif7.0 external-ids:"xs-network-uuid"="aee1ff9c-251d-817b-e6b6-128aed8e74fd"
Dec 16 09:40:58 xen-02 scripts-vif: vif7.0 external-ids:"attached-mac"="ce:1c:28:04:ff:c5"
Dec 16 09:40:58 xen-02 ha-lizard: 5140 Spawning new instance of ha-lizard
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7439] - common.py: Detected network backend as: 'openvswitch'
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7439] - Called with interface=vif7.0, uuid=174112ca-28d4-bbfe-6036-a2ee7a34e444, devid=0, network_mode=openvswitch, action=filter
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7439] - ['ovs-vsctl', 'iface-to-br', 'vif7.0']
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7439] - ['ovs-vsctl', 'br-to-parent', 'xenbr5']
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7439] - ['/sbin/ip', 'link', 'set', 'vif7.0', 'down']
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7439] - ['ovs-vsctl', 'get', 'interface', 'vif7.0', 'ofport']
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7439] - ['ovs-ofctl', 'del-flows', 'xenbr5', 'in_port=10']
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7439] - Got locking config: {'ipv4_allowed': [], 'mac': u'ce:1c:28:04:ff:c5', 'ipv6_allowed': [], 'locking_mode': u'unlocked'}
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7439] - ['/sbin/ip', 'link', 'set', 'vif7.0', 'up']
Dec 16 09:40:58 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:58 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:58 xen-02 ha-lizard:  This iteration is count 4479
Dec 16 09:40:58 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:58 xen-02 script-vif: vif7.0: writing backend/vif/7/0/hotplug-status=connected
Dec 16 09:40:58 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:40:58 xen-02 vif-real: Called as: /usr/libexec/xenopsd/vif-real tap 7 0 add
Dec 16 09:40:58 xen-02 scripts-vif: Called as "tap 7 0 add" domid:7 devid:0 mode:openvswitch
Dec 16 09:40:58 xen-02 scripts-vif: Failed to read backend/vif/7/0/bridge from xenstore. Trying /xapi/174112ca-28d4-bbfe-6036-a2ee7a34e444/private/vif/0/bridge
Dec 16 09:40:58 xen-02 scripts-vif: Adding tap7.0 to xenbr5 with address fe:ff:ff:ff:ff:ff
Dec 16 09:40:58 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:40:58 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:40:58 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:40:58 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:40:58 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7593] - common.py: Detected network backend as: 'openvswitch'
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7593] - Called with interface=tap7.0, uuid=174112ca-28d4-bbfe-6036-a2ee7a34e444, devid=0, network_mode=openvswitch, action=filter
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7593] - ['ovs-vsctl', 'iface-to-br', 'tap7.0']
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7593] - ['ovs-vsctl', 'br-to-parent', 'xenbr5']
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7593] - ['/sbin/ip', 'link', 'set', 'tap7.0', 'down']
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7593] - ['ovs-vsctl', 'get', 'interface', 'tap7.0', 'ofport']
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7593] - ['ovs-ofctl', 'del-flows', 'xenbr5', 'in_port=11']
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7593] - Got locking config: {'ipv4_allowed': [], 'mac': u'ce:1c:28:04:ff:c5', 'ipv6_allowed': [], 'locking_mode': u'unlocked'}
Dec 16 09:40:58 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[7593] - ['/sbin/ip', 'link', 'set', 'tap7.0', 'up']
Dec 16 09:40:59 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:40:59 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:40:59 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:40:59 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:41:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:41:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:41:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:41:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:41:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:41:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] validaton start
Dec 16 09:41:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:41:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] Validation passed
Dec 16 09:41:00 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:41:00 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:00 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:00 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:00 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:41:00 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:41:00 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:41:00 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:41:00 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:41:00 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:00 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:01 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:01 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:01 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:01 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:41:01 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:01 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:41:01 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:41:01 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:41:03 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:05 xen-02 iscsi-ha: 6965 Spawning new instance of iscsi-ha
Dec 16 09:41:05 xen-02 iscsi-ha: 6965 check_logger_processes Checking logger processes
Dec 16 09:41:05 xen-02 iscsi-ha: 6965 check_logger_processes No processes to clear
Dec 16 09:41:05 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:41:05 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:41:05 xen-02 iscsi-ha:  This iteration is count 6682
Dec 16 09:41:05 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:05 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:41:05 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:41:05 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:41:05 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:41:05 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:41:05 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:41:05 xen-02 iscsi-ha: 8282 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:41:05 xen-02 iscsi-ha: 8282 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:41:05 xen-02 iscsi-ha: 8282 service_execute: systemctl mode being used
Dec 16 09:41:05 xen-02 iscsi-ha: 8282 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:41:05 xen-02 iscsi-ha: 8282 service_execute: Returning exit status [ 3 ]
Dec 16 09:41:05 xen-02 iscsi-ha: 8282 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:41:05 xen-02 iscsi-ha: 8282 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485592 dw:1087485592 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:41:05 xen-02 iscsi-ha: 8282 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:41:05 xen-02 iscsi-ha: 8282 DRBD Resource: iscsi1 in Connected state
Dec 16 09:41:05 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:05 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:08 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:08 xen-02 vif-real: Called as: /usr/libexec/xenopsd/vif-real tap 7 0 remove
Dec 16 09:41:08 xen-02 scripts-vif: Called as "tap 7 0 remove" domid:7 devid:0 mode:openvswitch
Dec 16 09:41:08 xen-02 scripts-vif: tap7.0 has been removed
Dec 16 09:41:08 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[8431] - common.py: Detected network backend as: 'openvswitch'
Dec 16 09:41:08 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[8431] - Called with interface=tap7.0, uuid=174112ca-28d4-bbfe-6036-a2ee7a34e444, devid=0, network_mode=openvswitch, action=clear
Dec 16 09:41:08 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[8431] - ['ovs-vsctl', 'iface-to-br', 'tap7.0']
Dec 16 09:41:08 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[8431] - ['ovs-vsctl', 'br-to-parent', 'xenbr5']
Dec 16 09:41:08 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[8431] - ['/sbin/ip', 'link', 'set', 'tap7.0', 'down']
Dec 16 09:41:08 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[8431] - ['ovs-vsctl', 'get', 'interface', 'tap7.0', 'ofport']
Dec 16 09:41:08 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[8431] - ['ovs-ofctl', 'del-flows', 'xenbr5', 'in_port=11']
Dec 16 09:41:08 xen-02 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[8431] - ['/sbin/ip', 'link', 'set', 'tap7.0', 'up']
Dec 16 09:41:13 xen-02 ha-lizard: 7459 Spawning new instance of ha-lizard
Dec 16 09:41:13 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:13 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:41:13 xen-02 ha-lizard:  This iteration is count 4480
Dec 16 09:41:13 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:13 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:41:13 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:41:13 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:41:13 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:41:13 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:41:14 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:41:15 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:41:15 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:41:15 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:41:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:41:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:41:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:41:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:41:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:41:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:41:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] validaton start
Dec 16 09:41:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:41:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] Validation passed
Dec 16 09:41:15 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:41:15 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:15 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:15 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:15 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:41:15 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:41:15 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:41:15 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:41:15 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:41:15 xen-02 iscsi-ha: 8245 Spawning new instance of iscsi-ha
Dec 16 09:41:15 xen-02 iscsi-ha: 8245 check_logger_processes Checking logger processes
Dec 16 09:41:15 xen-02 iscsi-ha: 8245 check_logger_processes No processes to clear
Dec 16 09:41:15 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:41:15 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:41:15 xen-02 iscsi-ha:  This iteration is count 6683
Dec 16 09:41:15 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:41:15 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:15 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:41:15 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:41:15 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:41:15 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:41:15 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:41:15 xen-02 iscsi-ha: 8768 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:41:15 xen-02 iscsi-ha: 8768 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:41:15 xen-02 iscsi-ha: 8768 service_execute: systemctl mode being used
Dec 16 09:41:15 xen-02 iscsi-ha: 8768 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:41:15 xen-02 iscsi-ha: 8768 service_execute: Returning exit status [ 3 ]
Dec 16 09:41:15 xen-02 iscsi-ha: 8768 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:41:15 xen-02 iscsi-ha: 8768 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485592 dw:1087485592 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:41:15 xen-02 iscsi-ha: 8768 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:41:15 xen-02 iscsi-ha: 8768 DRBD Resource: iscsi1 in Connected state
Dec 16 09:41:15 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:16 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:16 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:16 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:16 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:16 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:16 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:16 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:41:16 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:16 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:41:16 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:41:16 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:41:18 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:23 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:25 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:41:25 xen-02 iscsi-ha: 8698 Spawning new instance of iscsi-ha
Dec 16 09:41:25 xen-02 iscsi-ha: 8698 check_logger_processes Checking logger processes
Dec 16 09:41:25 xen-02 iscsi-ha: 8698 check_logger_processes No processes to clear
Dec 16 09:41:25 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:41:25 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:41:25 xen-02 iscsi-ha:  This iteration is count 6684
Dec 16 09:41:25 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:25 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:41:25 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:41:25 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:41:26 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:41:26 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:41:26 xen-02 iscsi-ha: 9265 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:41:26 xen-02 iscsi-ha: 9265 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:41:26 xen-02 iscsi-ha: 9265 service_execute: systemctl mode being used
Dec 16 09:41:26 xen-02 iscsi-ha: 9265 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:41:26 xen-02 iscsi-ha: 9265 service_execute: Returning exit status [ 3 ]
Dec 16 09:41:26 xen-02 iscsi-ha: 9265 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:41:26 xen-02 iscsi-ha: 9265 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485592 dw:1087485592 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:41:26 xen-02 iscsi-ha: 9265 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:41:26 xen-02 iscsi-ha: 9265 DRBD Resource: iscsi1 in Connected state
Dec 16 09:41:26 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:26 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:28 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:28 xen-02 ha-lizard: 8509 Spawning new instance of ha-lizard
Dec 16 09:41:28 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:41:28 xen-02 ha-lizard:  This iteration is count 4481
Dec 16 09:41:28 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:28 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:41:29 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:41:29 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:41:29 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:41:29 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:41:29 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:41:30 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:41:30 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:41:30 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:41:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:41:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:41:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:41:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:41:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:41:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:41:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] validaton start
Dec 16 09:41:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:41:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] Validation passed
Dec 16 09:41:30 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:41:30 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:30 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:30 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:30 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:41:30 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:41:30 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:41:30 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:41:30 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:41:31 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:31 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:31 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:31 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:31 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:31 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:41:31 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:32 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:41:32 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:41:32 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:41:33 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:35 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:41:35 xen-02 iscsi-ha: 9192 Spawning new instance of iscsi-ha
Dec 16 09:41:35 xen-02 iscsi-ha: 9192 check_logger_processes Checking logger processes
Dec 16 09:41:35 xen-02 iscsi-ha: 9192 check_logger_processes No processes to clear
Dec 16 09:41:36 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:41:36 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:41:36 xen-02 iscsi-ha:  This iteration is count 6685
Dec 16 09:41:36 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:36 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:41:36 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:41:36 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:41:36 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:41:36 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:41:36 xen-02 iscsi-ha: 11252 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:41:36 xen-02 iscsi-ha: 11252 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:41:36 xen-02 iscsi-ha: 11252 service_execute: systemctl mode being used
Dec 16 09:41:36 xen-02 iscsi-ha: 11252 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:41:36 xen-02 iscsi-ha: 11252 service_execute: Returning exit status [ 3 ]
Dec 16 09:41:36 xen-02 iscsi-ha: 11252 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:41:36 xen-02 iscsi-ha: 11252 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087485592 dw:1087485592 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:41:36 xen-02 iscsi-ha: 11252 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:41:36 xen-02 iscsi-ha: 11252 DRBD Resource: iscsi1 in Connected state
Dec 16 09:41:36 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:36 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:38 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:43 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:43 xen-02 ha-lizard: 9957 Spawning new instance of ha-lizard
Dec 16 09:41:43 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:41:44 xen-02 ha-lizard:  This iteration is count 4482
Dec 16 09:41:44 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:44 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:41:44 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:41:44 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:41:44 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:41:44 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:41:44 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:41:45 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:41:45 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:41:45 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:41:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:41:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:41:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:41:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:41:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:41:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:41:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] validaton start
Dec 16 09:41:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:41:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] Validation passed
Dec 16 09:41:45 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:41:45 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:41:45 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:45 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:45 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:45 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:41:45 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:41:45 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:41:45 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:41:45 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:41:46 xen-02 iscsi-ha: 11217 Spawning new instance of iscsi-ha
Dec 16 09:41:46 xen-02 iscsi-ha: 11217 check_logger_processes Checking logger processes
Dec 16 09:41:46 xen-02 iscsi-ha: 11217 check_logger_processes No processes to clear
Dec 16 09:41:46 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:41:46 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:41:46 xen-02 iscsi-ha:  This iteration is count 6686
Dec 16 09:41:46 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:46 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:41:46 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:41:46 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:41:46 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:46 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:41:46 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:41:46 xen-02 iscsi-ha: 11681 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:41:46 xen-02 iscsi-ha: 11681 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:41:46 xen-02 iscsi-ha: 11681 service_execute: systemctl mode being used
Dec 16 09:41:46 xen-02 iscsi-ha: 11681 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:41:46 xen-02 iscsi-ha: 11681 service_execute: Returning exit status [ 3 ]
Dec 16 09:41:46 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:46 xen-02 iscsi-ha: 11681 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:41:46 xen-02 iscsi-ha: 11681 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087489868 dw:1087489868 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:41:46 xen-02 iscsi-ha: 11681 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:41:46 xen-02 iscsi-ha: 11681 DRBD Resource: iscsi1 in Connected state
Dec 16 09:41:46 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:46 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:46 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:46 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:41:47 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:47 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:41:47 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:41:47 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:41:48 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:51 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:51 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:53 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:55 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:41:56 xen-02 iscsi-ha: 11628 Spawning new instance of iscsi-ha
Dec 16 09:41:56 xen-02 iscsi-ha: 11628 check_logger_processes Checking logger processes
Dec 16 09:41:56 xen-02 iscsi-ha: 11628 check_logger_processes No processes to clear
Dec 16 09:41:56 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:41:56 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:41:56 xen-02 iscsi-ha:  This iteration is count 6687
Dec 16 09:41:56 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:56 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:41:56 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:41:56 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:41:56 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:41:56 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:41:56 xen-02 iscsi-ha: 12540 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:41:56 xen-02 iscsi-ha: 12540 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:41:56 xen-02 iscsi-ha: 12540 service_execute: systemctl mode being used
Dec 16 09:41:56 xen-02 iscsi-ha: 12540 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:41:56 xen-02 iscsi-ha: 12540 service_execute: Returning exit status [ 3 ]
Dec 16 09:41:56 xen-02 iscsi-ha: 12540 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:41:56 xen-02 iscsi-ha: 12540 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087489868 dw:1087489868 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:41:56 xen-02 iscsi-ha: 12540 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:41:56 xen-02 iscsi-ha: 12540 DRBD Resource: iscsi1 in Connected state
Dec 16 09:41:56 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:56 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:58 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:59 xen-02 ha-lizard: 11364 Spawning new instance of ha-lizard
Dec 16 09:41:59 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:41:59 xen-02 ha-lizard:  This iteration is count 4483
Dec 16 09:41:59 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:59 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:41:59 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:41:59 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:41:59 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:41:59 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:41:59 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:42:00 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:42:00 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:42:00 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:42:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:42:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:42:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:42:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:42:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:42:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:42:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] validaton start
Dec 16 09:42:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:42:00 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] Validation passed
Dec 16 09:42:00 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:42:00 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:00 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:00 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:00 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:42:00 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:42:00 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:42:00 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:42:01 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:42:01 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:01 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:01 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:01 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:01 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:02 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:42:02 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:02 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:42:02 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:42:02 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:42:03 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:05 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:42:06 xen-02 iscsi-ha: 12505 Spawning new instance of iscsi-ha
Dec 16 09:42:06 xen-02 iscsi-ha: 12505 check_logger_processes Checking logger processes
Dec 16 09:42:06 xen-02 iscsi-ha: 12505 check_logger_processes No processes to clear
Dec 16 09:42:06 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:42:06 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:42:06 xen-02 iscsi-ha:  This iteration is count 6688
Dec 16 09:42:06 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:06 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:42:06 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:42:06 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:42:06 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:42:06 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:42:06 xen-02 iscsi-ha: 14198 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:42:06 xen-02 iscsi-ha: 14198 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:42:06 xen-02 iscsi-ha: 14198 service_execute: systemctl mode being used
Dec 16 09:42:06 xen-02 iscsi-ha: 14198 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:42:06 xen-02 iscsi-ha: 14198 service_execute: Returning exit status [ 3 ]
Dec 16 09:42:06 xen-02 iscsi-ha: 14198 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:42:06 xen-02 iscsi-ha: 14198 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087489868 dw:1087489868 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:42:06 xen-02 iscsi-ha: 14198 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:42:06 xen-02 iscsi-ha: 14198 DRBD Resource: iscsi1 in Connected state
Dec 16 09:42:06 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:06 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:08 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:13 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:14 xen-02 ha-lizard: 12636 Spawning new instance of ha-lizard
Dec 16 09:42:14 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:42:14 xen-02 ha-lizard:  This iteration is count 4484
Dec 16 09:42:14 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:14 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:42:14 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:42:14 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:42:14 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:42:14 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:42:14 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:42:15 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:42:15 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:42:15 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:42:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:42:15 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:42:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:42:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:42:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:42:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:42:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:42:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] validaton start
Dec 16 09:42:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:42:15 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] Validation passed
Dec 16 09:42:15 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:42:15 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:15 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:15 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:16 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:42:16 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:42:16 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:42:16 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:42:16 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:42:16 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:16 xen-02 iscsi-ha: 14127 Spawning new instance of iscsi-ha
Dec 16 09:42:16 xen-02 iscsi-ha: 14127 check_logger_processes Checking logger processes
Dec 16 09:42:16 xen-02 iscsi-ha: 14127 check_logger_processes No processes to clear
Dec 16 09:42:16 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:16 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:42:16 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:42:16 xen-02 iscsi-ha:  This iteration is count 6689
Dec 16 09:42:16 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:16 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:42:16 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:42:16 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:42:16 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:42:16 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:42:16 xen-02 iscsi-ha: 14894 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:42:16 xen-02 iscsi-ha: 14894 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:42:16 xen-02 iscsi-ha: 14894 service_execute: systemctl mode being used
Dec 16 09:42:16 xen-02 iscsi-ha: 14894 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:42:16 xen-02 iscsi-ha: 14894 service_execute: Returning exit status [ 3 ]
Dec 16 09:42:16 xen-02 iscsi-ha: 14894 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:42:16 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:16 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:16 xen-02 iscsi-ha: 14894 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087490072 dw:1087490072 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:42:16 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:16 xen-02 iscsi-ha: 14894 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:42:16 xen-02 iscsi-ha: 14894 DRBD Resource: iscsi1 in Connected state
Dec 16 09:42:17 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:17 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:17 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:42:17 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:17 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:42:17 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:42:17 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:42:18 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:23 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:25 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:42:26 xen-02 iscsi-ha: 14831 Spawning new instance of iscsi-ha
Dec 16 09:42:26 xen-02 iscsi-ha: 14831 check_logger_processes Checking logger processes
Dec 16 09:42:26 xen-02 iscsi-ha: 14831 check_logger_processes No processes to clear
Dec 16 09:42:26 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:42:26 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:42:26 xen-02 iscsi-ha:  This iteration is count 6690
Dec 16 09:42:26 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:26 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:42:26 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:42:26 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:42:26 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:42:26 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:42:27 xen-02 iscsi-ha: 15242 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:42:27 xen-02 iscsi-ha: 15242 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:42:27 xen-02 iscsi-ha: 15242 service_execute: systemctl mode being used
Dec 16 09:42:27 xen-02 iscsi-ha: 15242 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:42:27 xen-02 iscsi-ha: 15242 service_execute: Returning exit status [ 3 ]
Dec 16 09:42:27 xen-02 iscsi-ha: 15242 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:42:27 xen-02 iscsi-ha: 15242 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087490072 dw:1087490072 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:42:27 xen-02 iscsi-ha: 15242 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:42:27 xen-02 iscsi-ha: 15242 DRBD Resource: iscsi1 in Connected state
Dec 16 09:42:27 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:27 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:28 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:29 xen-02 ha-lizard: 14579 Spawning new instance of ha-lizard
Dec 16 09:42:29 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:42:29 xen-02 ha-lizard:  This iteration is count 4485
Dec 16 09:42:29 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:29 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:42:29 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:42:29 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:42:29 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:42:29 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:42:29 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:42:30 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:42:30 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:42:30 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:42:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:42:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:42:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:42:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:42:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:42:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:42:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] validaton start
Dec 16 09:42:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:42:30 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] Validation passed
Dec 16 09:42:30 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:42:31 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:31 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:31 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:31 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:42:31 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:42:31 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:42:31 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:42:31 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:42:31 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:31 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:32 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:32 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:32 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:32 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:42:32 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:32 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:42:32 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:42:32 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:42:33 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:35 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:42:36 xen-02 iscsi-ha: 15207 Spawning new instance of iscsi-ha
Dec 16 09:42:36 xen-02 iscsi-ha: 15207 check_logger_processes Checking logger processes
Dec 16 09:42:36 xen-02 iscsi-ha: 15207 check_logger_processes No processes to clear
Dec 16 09:42:36 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:42:36 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:42:36 xen-02 iscsi-ha:  This iteration is count 6691
Dec 16 09:42:36 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:37 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:42:37 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:42:37 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:42:37 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:42:37 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:42:37 xen-02 iscsi-ha: 15854 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:42:37 xen-02 iscsi-ha: 15854 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:42:37 xen-02 iscsi-ha: 15854 service_execute: systemctl mode being used
Dec 16 09:42:37 xen-02 iscsi-ha: 15854 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:42:37 xen-02 iscsi-ha: 15854 service_execute: Returning exit status [ 3 ]
Dec 16 09:42:37 xen-02 iscsi-ha: 15854 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:42:37 xen-02 iscsi-ha: 15854 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087490072 dw:1087490072 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:42:37 xen-02 iscsi-ha: 15854 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:42:37 xen-02 iscsi-ha: 15854 DRBD Resource: iscsi1 in Connected state
Dec 16 09:42:37 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:37 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:38 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:43 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:44 xen-02 ha-lizard: 15334 Spawning new instance of ha-lizard
Dec 16 09:42:44 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:42:44 xen-02 ha-lizard:  This iteration is count 4486
Dec 16 09:42:44 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:44 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:42:44 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:42:44 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:42:44 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:42:44 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:42:44 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:42:45 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:42:45 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:42:45 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:42:45 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:42:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:42:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:42:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:42:45 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:42:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:42:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:42:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] validaton start
Dec 16 09:42:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:42:46 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] Validation passed
Dec 16 09:42:46 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:42:46 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:46 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:46 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:46 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:42:46 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:42:46 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:42:46 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:42:46 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:42:46 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:46 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:47 xen-02 iscsi-ha: 15819 Spawning new instance of iscsi-ha
Dec 16 09:42:47 xen-02 iscsi-ha: 15819 check_logger_processes Checking logger processes
Dec 16 09:42:47 xen-02 iscsi-ha: 15819 check_logger_processes No processes to clear
Dec 16 09:42:47 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:42:47 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:42:47 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:47 xen-02 iscsi-ha:  This iteration is count 6692
Dec 16 09:42:47 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:47 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:47 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:47 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:42:47 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:42:47 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:42:47 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:42:47 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:42:47 xen-02 iscsi-ha: 17634 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:42:47 xen-02 iscsi-ha: 17634 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:42:47 xen-02 iscsi-ha: 17634 service_execute: systemctl mode being used
Dec 16 09:42:47 xen-02 iscsi-ha: 17634 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:42:47 xen-02 iscsi-ha: 17634 service_execute: Returning exit status [ 3 ]
Dec 16 09:42:47 xen-02 iscsi-ha: 17634 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:42:47 xen-02 iscsi-ha: 17634 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087490212 dw:1087490212 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:42:47 xen-02 iscsi-ha: 17634 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:42:47 xen-02 iscsi-ha: 17634 DRBD Resource: iscsi1 in Connected state
Dec 16 09:42:47 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:42:47 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:47 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:47 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:47 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:42:47 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:42:47 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:42:48 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:53 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:55 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:42:57 xen-02 iscsi-ha: 17567 Spawning new instance of iscsi-ha
Dec 16 09:42:57 xen-02 iscsi-ha: 17567 check_logger_processes Checking logger processes
Dec 16 09:42:57 xen-02 iscsi-ha: 17567 check_logger_processes No processes to clear
Dec 16 09:42:57 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:42:57 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:42:57 xen-02 iscsi-ha:  This iteration is count 6693
Dec 16 09:42:57 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:57 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:42:57 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:42:57 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:42:57 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:42:57 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:42:57 xen-02 iscsi-ha: 17946 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:42:57 xen-02 iscsi-ha: 17946 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:42:57 xen-02 iscsi-ha: 17946 service_execute: systemctl mode being used
Dec 16 09:42:57 xen-02 iscsi-ha: 17946 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:42:57 xen-02 iscsi-ha: 17946 service_execute: Returning exit status [ 3 ]
Dec 16 09:42:57 xen-02 iscsi-ha: 17946 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:42:57 xen-02 iscsi-ha: 17946 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087490212 dw:1087490212 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:42:57 xen-02 iscsi-ha: 17946 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:42:57 xen-02 iscsi-ha: 17946 DRBD Resource: iscsi1 in Connected state
Dec 16 09:42:57 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:57 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:58 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:59 xen-02 ha-lizard: 17279 Spawning new instance of ha-lizard
Dec 16 09:42:59 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:42:59 xen-02 ha-lizard:  This iteration is count 4487
Dec 16 09:42:59 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:59 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:42:59 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:42:59 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:42:59 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:42:59 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:42:59 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:43:00 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:43:00 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:43:00 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:43:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:43:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:43:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:43:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:43:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:43:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:43:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] validaton start
Dec 16 09:43:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:43:01 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] Validation passed
Dec 16 09:43:01 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:43:01 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:43:01 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:43:01 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:43:01 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:43:01 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:43:01 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:43:01 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:43:01 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:43:01 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:43:02 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:43:02 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:43:02 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:43:02 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:43:02 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:43:02 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:43:02 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:43:02 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:43:02 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:43:03 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:43:05 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:43:07 xen-02 iscsi-ha: 17911 Spawning new instance of iscsi-ha
Dec 16 09:43:07 xen-02 iscsi-ha: 17911 check_logger_processes Checking logger processes
Dec 16 09:43:07 xen-02 iscsi-ha: 17911 check_logger_processes No processes to clear
Dec 16 09:43:07 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:43:07 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:43:07 xen-02 iscsi-ha:  This iteration is count 6694
Dec 16 09:43:07 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:43:07 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:43:07 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:43:07 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:43:07 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:43:07 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:43:07 xen-02 iscsi-ha: 18561 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:43:07 xen-02 iscsi-ha: 18561 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:43:07 xen-02 iscsi-ha: 18561 service_execute: systemctl mode being used
Dec 16 09:43:07 xen-02 iscsi-ha: 18561 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:43:07 xen-02 iscsi-ha: 18561 service_execute: Returning exit status [ 3 ]
Dec 16 09:43:07 xen-02 iscsi-ha: 18561 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:43:07 xen-02 iscsi-ha: 18561 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087490212 dw:1087490212 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:43:07 xen-02 iscsi-ha: 18561 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:43:07 xen-02 iscsi-ha: 18561 DRBD Resource: iscsi1 in Connected state
Dec 16 09:43:07 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:43:07 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:43:08 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:43:13 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:43:14 xen-02 ha-lizard: 18035 Spawning new instance of ha-lizard
Dec 16 09:43:14 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:43:14 xen-02 ha-lizard:  This iteration is count 4488
Dec 16 09:43:14 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:43:14 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
Dec 16 09:43:15 xen-02 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:43:15 xen-02 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:43:15 xen-02 ha-lizard:  master_ip: Pool Master IP Address =  172.16.14.199
Dec 16 09:43:15 xen-02 ha-lizard:  Validating master is still a master
Dec 16 09:43:15 xen-02 ha-lizard:  [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.16.14.199 ]
Dec 16 09:43:15 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:43:16 xen-02 ha-lizard:  This slave - xen-02: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1
Dec 16 09:43:16 xen-02 ha-lizard:  check_xapi: Pool Host 172.16.14.199 xapi status = 13
Dec 16 09:43:16 xen-02 ha-lizard:  validate_this_host_vm_states: MY HOST UUID = [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] - validated
Dec 16 09:43:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] validaton start
Dec 16 09:43:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:43:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ a6c71084-7bb3-4f8e-b700-d1ba368d260a ] Validation passed
Dec 16 09:43:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] validaton start
Dec 16 09:43:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:43:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] Validation passed
Dec 16 09:43:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] validaton start
Dec 16 09:43:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] reported as running on host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ]
Dec 16 09:43:16 xen-02 ha-lizard:  validate_this_host_vm_states: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] Validation passed
Dec 16 09:43:16 xen-02 ha-lizard:  Pool Master is OK - calling function check_ha_enabled - updating local status
Dec 16 09:43:16 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:43:16 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:43:16 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:43:16 xen-02 ha-lizard:  Checking state file for status if ha-lizard is enabled
Dec 16 09:43:16 xen-02 ha-lizard:  Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled
Dec 16 09:43:16 xen-02 ha-lizard:  ha-lizard is enabled - continuing
Dec 16 09:43:16 xen-02 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:43:16 xen-02 ha-lizard:  Calling Function write_pool_state - updating local state files
Dec 16 09:43:17 xen-02 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:43:17 xen-02 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:43:17 xen-02 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:43:17 xen-02 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:43:17 xen-02 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:43:17 xen-02 iscsi-ha: 18519 Spawning new instance of iscsi-ha
Dec 16 09:43:17 xen-02 iscsi-ha: 18519 check_logger_processes Checking logger processes
Dec 16 09:43:17 xen-02 iscsi-ha: 18519 check_logger_processes No processes to clear
Dec 16 09:43:17 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:43:17 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:43:17 xen-02 iscsi-ha:  This iteration is count 6695
Dec 16 09:43:17 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:43:17 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:43:17 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:43:17 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:43:17 xen-02 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:43:17 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:43:17 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:43:17 xen-02 iscsi-ha: 20254 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:43:17 xen-02 iscsi-ha: 20254 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:43:17 xen-02 iscsi-ha: 20254 service_execute: systemctl mode being used
Dec 16 09:43:17 xen-02 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:43:17 xen-02 iscsi-ha: 20254 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:43:17 xen-02 iscsi-ha: 20254 service_execute: Returning exit status [ 3 ]
Dec 16 09:43:17 xen-02 iscsi-ha: 20254 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:43:17 xen-02 iscsi-ha: 20254 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087490292 dw:1087490292 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:43:17 xen-02 iscsi-ha: 20254 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:43:17 xen-02 iscsi-ha: 20254 DRBD Resource: iscsi1 in Connected state
Dec 16 09:43:17 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:43:17 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:43:17 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:43:18 xen-02 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:43:18 xen-02 ha-lizard:  write_status_report: Writing status report
Dec 16 09:43:18 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:43:23 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:43:25 xen-02 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:43:27 xen-02 iscsi-ha: 20137 Spawning new instance of iscsi-ha
Dec 16 09:43:27 xen-02 iscsi-ha: 20137 check_logger_processes Checking logger processes
Dec 16 09:43:27 xen-02 iscsi-ha: 20137 check_logger_processes No processes to clear
Dec 16 09:43:27 xen-02 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:43:27 xen-02 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:43:27 xen-02 iscsi-ha:  This iteration is count 6696
Dec 16 09:43:27 xen-02 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:43:27 xen-02 iscsi-ha:  This host's pool status = slave:172.16.14.199
Dec 16 09:43:27 xen-02 iscsi-ha:  service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:43:27 xen-02 iscsi-ha:  service_execute: System V mode detected
Dec 16 09:43:27 xen-02 iscsi-ha:  service_execute: [  OK  ]#015iscsi-ha running: 4065
Dec 16 09:43:27 xen-02 iscsi-ha:  service_execute: Returning exit status [ 0 ]
Dec 16 09:43:27 xen-02 iscsi-ha: 20654 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.200#012172.16.14.49#01210.10.10.2
Dec 16 09:43:27 xen-02 iscsi-ha: 20654 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:43:27 xen-02 iscsi-ha: 20654 service_execute: systemctl mode being used
Dec 16 09:43:27 xen-02 iscsi-ha: 20654 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: inactive (dead)
Dec 16 09:43:27 xen-02 iscsi-ha: 20654 service_execute: Returning exit status [ 3 ]
Dec 16 09:43:28 xen-02 iscsi-ha: 20654 iSCSI target: tgtd status stopped. Expected Stopped . [inactive (dead)]
Dec 16 09:43:28 xen-02 iscsi-ha: 20654 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101)#012srcversion: FB3AC7056350AC64629E395 #012#012 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----#012    ns:0 nr:1087490292 dw:1087490292 dr:0 al:0 bm:197372 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:43:28 xen-02 iscsi-ha: 20654 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Dec 16 09:43:28 xen-02 iscsi-ha: 20654 DRBD Resource: iscsi1 in Connected state
Dec 16 09:43:28 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:43:28 xen-02 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:43:28 xen-02 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:43:29 xen-02 ha-lizard: 19226 Spawning new instance of ha-lizard
Dec 16 09:43:29 xen-02 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:43:29 xen-02 ha-lizard:  This iteration is count 4489
Dec 16 09:43:29 xen-02 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:43:29 xen-02 ha-lizard:  This host's pool status = slave:172.16.14.199
