Dec 16 09:32:30 xen-01 ha-lizard: 9091 autoselect_slave: MASTER host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:32:30 xen-01 ha-lizard:  get_vms_on_host: Returned 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:32:30 xen-01 ha-lizard: 9097 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:32:30 xen-01 ha-lizard: 9091 autoselect_slave: 28dfaba4-5b90-4535-afec-a244a6c3f88a is Master UUID - excluding from list of available slaves
Dec 16 09:32:30 xen-01 ha-lizard: 9097 check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:32:30 xen-01 ha-lizard: 9097 check_slave_status: Management link OK - continue
Dec 16 09:32:30 xen-01 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:32:30 xen-01 ha-lizard: 9097 get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:32:30 xen-01 ha-lizard: 9091 autoselect_slave: 1 available Slave UUIDs found: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_slave_status: Removing Master UUID from list of Hosts
Dec 16 09:32:31 xen-01 ha-lizard: 9091 autoselect_slave: Selected Slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce = Current slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce - ignoring update
Dec 16 09:32:31 xen-01 ha-lizard: 9097 get_pool_ip_list: returned 172.16.14.200
Dec 16 09:32:31 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:32:31 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:32:31 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_xapi: Pool Host 172.16.14.200 xapi status = 13
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_slave_status: Host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] marked as healthy
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_slave_status: Host IP Address check Status Array for Slaves = (1)
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_slave_status: Quorum check called
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_quorum: Checking host IPs: 172.16.14.199 172.16.14.200
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_quorum: Host IP: 172.16.14.199 Response = OK
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_quorum: LIVE HOSTs = 1
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_quorum: Host IP: 172.16.14.200 Response = OK
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_quorum: LIVE HOSTs = 2
Dec 16 09:32:31 xen-01 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_quorum: Using network points: 172.16.14.165 as possible additional vote
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_quorum: Heuristic IP: 172.16.14.165 Response = OK
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_quorum: Successful Replies = 1
Dec 16 09:32:31 xen-01 ha-lizard: 9097 Total enpoints checked = 1 with total successful replies = 1
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_quorum: Additional heuristic vote success. Incremeting vote by 1
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed
Dec 16 09:32:31 xen-01 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_slave_status: Failed slave count = 0
Dec 16 09:32:31 xen-01 ha-lizard: 9097 check_slave_status: No Failed slaves detected
Dec 16 09:32:31 xen-01 ha-lizard: 9097 Function check_slave_status reported no failures: calling vm_mon
Dec 16 09:32:31 xen-01 ha-lizard: 9097 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Dec 16 09:32:31 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:32:31 xen-01 ha-lizard: 9097 vm_mon: Retrived list of VMs for this poll: 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:32:31 xen-01 ha-lizard: 9097 vm_mon: Removing Control Domains from VM list
Dec 16 09:32:31 xen-01 ha-lizard: 9097 vm_mon: VM list returned = 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:32:31 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:32:31 xen-01 ha-lizard:  write_status_report: Writing status report
Dec 16 09:32:31 xen-01 ha-lizard: 9097 vm_state: Machine state for 4c4ca253-ff4f-2702-94a2-836fc0380f31 returned: running
Dec 16 09:32:31 xen-01 ha-lizard: 9097 vm_mon: VM 4c4ca253-ff4f-2702-94a2-836fc0380f31 state = running
Dec 16 09:32:31 xen-01 ha-lizard: 9097 vm_state: Machine state for 174112ca-28d4-bbfe-6036-a2ee7a34e444 returned: running
Dec 16 09:32:31 xen-01 ha-lizard: 9097 vm_mon: VM 174112ca-28d4-bbfe-6036-a2ee7a34e444 state = running
Dec 16 09:32:31 xen-01 ha-lizard: 9097 vm_mon: 0 Eligible Halted VMs found
Dec 16 09:32:32 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:32:33 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:38 xen-01 iscsi-ha: 8437 Spawning new instance of iscsi-ha
Dec 16 09:32:38 xen-01 iscsi-ha: 8437 check_logger_processes Checking logger processes
Dec 16 09:32:38 xen-01 iscsi-ha: 8437 check_logger_processes No processes to clear
Dec 16 09:32:38 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:32:38 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:32:38 xen-01 iscsi-ha:  This iteration is count 6932
Dec 16 09:32:38 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:38 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 service_execute: System V mode detected
Dec 16 09:32:38 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 service_execute: Returning exit status [ 0 ]
Dec 16 09:32:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431632 nr:3152 dw:4369688 dr:1325207884 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 DRBD Resource: iscsi1 in Connected state
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 service_execute: systemctl mode being used
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 service_execute: Returning exit status [ 0 ]
Dec 16 09:32:38 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 CHECKING IP 127.0.0.1
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 CHECKING IP 172.16.14.199
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 CHECKING IP 172.16.14.47
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 CHECKING IP 10.10.10.1
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 CHECKING IP 10.10.10.3
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:32:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:32:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:32:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:32:38 xen-01 iscsi-ha: 9653 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:32:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:32:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:32:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:32:42 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:32:43 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:45 xen-01 ha-lizard: 8891 Spawning new instance of ha-lizard
Dec 16 09:32:45 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:32:45 xen-01 ha-lizard:  This iteration is count 4629
Dec 16 09:32:45 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:45 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:32:45 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:32:45 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:32:45 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:32:45 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:32:45 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:32:45 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:32:45 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:32:45 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:32:45 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:32:45 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:32:45 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:32:45 xen-01 ha-lizard:  Master management link = DOWN
Dec 16 09:32:45 xen-01 ha-lizard:  TIMENOW = 1481880765
Dec 16 09:32:45 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:32:45 xen-01 ha-lizard:  TIMEELAPSED = 0
Dec 16 09:32:45 xen-01 ha-lizard:  MGT link failure duration = [ 0 ] Tolerance = [ 5 seconds ]
Dec 16 09:32:45 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:32:48 xen-01 iscsi-ha: 9637 Spawning new instance of iscsi-ha
Dec 16 09:32:48 xen-01 iscsi-ha: 9637 check_logger_processes Checking logger processes
Dec 16 09:32:48 xen-01 iscsi-ha: 9637 check_logger_processes No processes to clear
Dec 16 09:32:48 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:32:48 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:32:48 xen-01 iscsi-ha:  This iteration is count 6933
Dec 16 09:32:48 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:48 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 service_execute: System V mode detected
Dec 16 09:32:48 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:48 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 service_execute: Returning exit status [ 0 ]
Dec 16 09:32:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431640 nr:3152 dw:4369696 dr:1325207900 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 DRBD Resource: iscsi1 in Connected state
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 service_execute: systemctl mode being used
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 service_execute: Returning exit status [ 0 ]
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 CHECKING IP 127.0.0.1
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 CHECKING IP 172.16.14.199
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 CHECKING IP 172.16.14.47
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 CHECKING IP 10.10.10.1
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 CHECKING IP 10.10.10.3
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:32:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:32:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:32:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:32:48 xen-01 iscsi-ha: 10261 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:32:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:32:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:32:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:32:50 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:32:50 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:32:50 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:32:50 xen-01 ha-lizard:  TIMENOW = 1481880770
Dec 16 09:32:50 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:32:50 xen-01 ha-lizard:  TIMEELAPSED = 5
Dec 16 09:32:50 xen-01 ha-lizard:  MGT link failure duration = [ 5 ] Tolerance = [ 5 seconds ]
Dec 16 09:32:50 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:32:52 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:32:53 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:55 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:32:55 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:32:55 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:32:55 xen-01 ha-lizard:  TIMENOW = 1481880775
Dec 16 09:32:55 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:32:55 xen-01 ha-lizard:  TIMEELAPSED = 10
Dec 16 09:32:55 xen-01 ha-lizard:  MGT link failure duration = [ 10 ] Tolerance = [ 5 seconds ]
Dec 16 09:32:55 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:32:55 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:32:56 xen-01 ha-lizard:  stop_vms_on_host:: Destroying domain [ 7 ] VM [ CentOS 7 (2) ]
Dec 16 09:32:56 xen-01 ha-lizard:  stop_vms_on_host: Using XL toolstack mode [ /sbin/xl destroy 7 ]
Dec 16 09:32:56 xen-01 vif-real: Called as: /usr/libexec/xenopsd/vif-real vif 7 0 offline
Dec 16 09:32:56 xen-01 scripts-vif: Called as "vif 7 0 offline" domid:7 devid:0 mode:openvswitch
Dec 16 09:32:56 xen-01 vif-real: Called as: /usr/libexec/xenopsd/vif-real vif 7 0 remove
Dec 16 09:32:56 xen-01 scripts-block: backend/vbd3/7/5696: remove
Dec 16 09:32:56 xen-01 scripts-block: backend/vbd3/7/768: remove
Dec 16 09:32:56 xen-01 scripts-vif: Called as "vif 7 0 remove" domid:7 devid:0 mode:openvswitch
Dec 16 09:32:56 xen-01 script-vif: vif7.0: removing backend/vif/7/0/hotplug-status
Dec 16 09:32:56 xen-01 script-vif: vif7.0: removing /xapi/174112ca-28d4-bbfe-6036-a2ee7a34e444/hotplug/7/vif/0/hotplug
Dec 16 09:32:56 xen-01 scripts-vif: vif7.0 has been removed
Dec 16 09:32:56 xen-01 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[10654] - common.py: Detected network backend as: 'openvswitch'
Dec 16 09:32:56 xen-01 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[10654] - Called with interface=vif7.0, uuid=174112ca-28d4-bbfe-6036-a2ee7a34e444, devid=0, network_mode=openvswitch, action=clear
Dec 16 09:32:56 xen-01 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[10654] - ['ovs-vsctl', 'iface-to-br', 'vif7.0']
Dec 16 09:32:56 xen-01 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[10654] - ['ovs-vsctl', 'br-to-parent', 'xenbr5']
Dec 16 09:32:56 xen-01 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[10654] - ['/sbin/ip', 'link', 'set', 'vif7.0', 'down']
Dec 16 09:32:56 xen-01 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[10654] - ['ovs-vsctl', 'get', 'interface', 'vif7.0', 'ofport']
Dec 16 09:32:56 xen-01 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[10654] - ['ovs-ofctl', 'del-flows', 'xenbr5', 'in_port=11']
Dec 16 09:32:56 xen-01 setup-vif-rules: /usr/libexec/xenopsd/setup-vif-rules[10654] - ['/sbin/ip', 'link', 'set', 'vif7.0', 'up']
Dec 16 09:32:56 xen-01 ha-lizard:  stop_vms_on_host: Domain [ 7 ] [ CentOS 7 (2) ] power-state = [ paused ]
Dec 16 09:32:56 xen-01 ha-lizard:  domain [ 7 ] destroyed
Dec 16 09:32:56 xen-01 ha-lizard:  Resetting VDI for VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] [ CentOS 7 (2) ]
Dec 16 09:32:56 xen-01 ha-lizard:  reset_vm_vdi: Resetting VDI(s) for VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ]
Dec 16 09:32:56 xen-01 ha-lizard:  reset_vm_vdi: No VDI found for VBD [ 83c80601-1918-22bf-5cd3-cc21781c6c9e ]
Dec 16 09:32:56 xen-01 ha-lizard:  reset_vm_vdi: Found VDI [ 3d66fc17-d503-4bf4-aad8-7a4619a8425a ]
Dec 16 09:32:56 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Force-cleared RW for 3d66fc17-d503-4bf4-aad8-7a4619a8425a on host 28dfaba4-5b90-4535-afec-a244a6c3f88a (xen-01)
Dec 16 09:32:56 xen-01 ha-lizard:  reset_vm_vdi: VDI [ 3d66fc17-d503-4bf4-aad8-7a4619a8425a ] reset
Dec 16 09:32:56 xen-01 ha-lizard:  stop_vms_on_host: VDI reset success
Dec 16 09:32:56 xen-01 ha-lizard:  stop_vms_on_host: Reseting power-state for [ CentOS 7 (2) ]
Dec 16 09:32:57 xen-01 ha-lizard:  stop_vms_on_host: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] [ CentOS 7 (2) ] power-state = [ halted ]
Dec 16 09:32:57 xen-01 ha-lizard:  stop_vms_on_host: Destroy domain [ 7 ] UUID:[ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] [ CentOS 7 (2) ] success
Dec 16 09:32:57 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:32:58 xen-01 iscsi-ha: 10247 Spawning new instance of iscsi-ha
Dec 16 09:32:58 xen-01 iscsi-ha: 10247 check_logger_processes Checking logger processes
Dec 16 09:32:58 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:32:58 xen-01 iscsi-ha: 10247 check_logger_processes No processes to clear
Dec 16 09:32:58 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:32:58 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:32:58 xen-01 iscsi-ha:  This iteration is count 6934
Dec 16 09:32:58 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:32:58 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 service_execute: System V mode detected
Dec 16 09:32:58 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 service_execute: Returning exit status [ 0 ]
Dec 16 09:32:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431640 nr:3152 dw:4369696 dr:1325207916 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 DRBD Resource: iscsi1 in Connected state
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 service_execute: systemctl mode being used
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 service_execute: Returning exit status [ 0 ]
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 CHECKING IP 127.0.0.1
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 CHECKING IP 172.16.14.199
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 CHECKING IP 172.16.14.47
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 CHECKING IP 10.10.10.1
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 CHECKING IP 10.10.10.3
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:32:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:32:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:32:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:32:58 xen-01 iscsi-ha: 10788 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:32:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:32:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:32:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:33:00 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 1 on PIDS: 9831
Dec 16 09:33:02 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:33:02 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:33:02 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:33:02 xen-01 ha-lizard:  TIMENOW = 1481880782
Dec 16 09:33:02 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:33:02 xen-01 ha-lizard:  TIMEELAPSED = 17
Dec 16 09:33:02 xen-01 ha-lizard:  MGT link failure duration = [ 17 ] Tolerance = [ 5 seconds ]
Dec 16 09:33:02 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:02 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:02 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:33:02 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:33:03 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:04 xen-01 cmanicd: NIC agent link is down : eth4
Dec 16 09:33:07 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:33:07 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:33:07 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:33:07 xen-01 ha-lizard:  TIMENOW = 1481880787
Dec 16 09:33:07 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:33:07 xen-01 ha-lizard:  TIMEELAPSED = 22
Dec 16 09:33:07 xen-01 ha-lizard:  MGT link failure duration = [ 22 ] Tolerance = [ 5 seconds ]
Dec 16 09:33:07 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:07 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:07 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:33:08 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:08 xen-01 iscsi-ha: 10774 Spawning new instance of iscsi-ha
Dec 16 09:33:08 xen-01 iscsi-ha: 10774 check_logger_processes Checking logger processes
Dec 16 09:33:08 xen-01 iscsi-ha: 10774 check_logger_processes No processes to clear
Dec 16 09:33:08 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:33:08 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:33:08 xen-01 iscsi-ha:  This iteration is count 6935
Dec 16 09:33:08 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:08 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 service_execute: System V mode detected
Dec 16 09:33:08 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 service_execute: Returning exit status [ 0 ]
Dec 16 09:33:08 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:08 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431640 nr:3152 dw:4369696 dr:1325207932 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 DRBD Resource: iscsi1 in Connected state
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 service_execute: systemctl mode being used
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 service_execute: Returning exit status [ 0 ]
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 CHECKING IP 127.0.0.1
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 CHECKING IP 172.16.14.199
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 CHECKING IP 172.16.14.47
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 CHECKING IP 10.10.10.1
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 CHECKING IP 10.10.10.3
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:33:08 xen-01 iscsi-ha: 12044 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:33:09 xen-01 iscsi-ha: 12044 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:33:09 xen-01 iscsi-ha: 12044 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:33:09 xen-01 iscsi-ha: 12044 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:33:09 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:33:09 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:33:09 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:33:09 xen-01 iscsi-ha: 12044 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:33:09 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:33:09 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:33:09 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:33:10 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 2 on PIDS: 9831
Dec 16 09:33:12 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:33:12 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:33:12 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:33:12 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:33:12 xen-01 ha-lizard:  TIMENOW = 1481880792
Dec 16 09:33:12 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:33:12 xen-01 ha-lizard:  TIMEELAPSED = 27
Dec 16 09:33:12 xen-01 ha-lizard:  MGT link failure duration = [ 27 ] Tolerance = [ 5 seconds ]
Dec 16 09:33:12 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:12 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:12 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:33:13 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:17 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:33:17 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:33:17 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:33:17 xen-01 ha-lizard:  TIMENOW = 1481880797
Dec 16 09:33:17 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:33:17 xen-01 ha-lizard:  TIMEELAPSED = 32
Dec 16 09:33:17 xen-01 ha-lizard:  MGT link failure duration = [ 32 ] Tolerance = [ 5 seconds ]
Dec 16 09:33:17 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:17 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:18 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:33:18 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:18 xen-01 iscsi-ha: 12030 Spawning new instance of iscsi-ha
Dec 16 09:33:18 xen-01 iscsi-ha: 12030 check_logger_processes Checking logger processes
Dec 16 09:33:18 xen-01 iscsi-ha: 12030 check_logger_processes No processes to clear
Dec 16 09:33:18 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:33:18 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:33:18 xen-01 iscsi-ha:  This iteration is count 6936
Dec 16 09:33:18 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:18 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:33:18 xen-01 iscsi-ha: 12593 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:33:18 xen-01 iscsi-ha: 12593 service_execute: System V mode detected
Dec 16 09:33:18 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:18 xen-01 iscsi-ha: 12593 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:33:18 xen-01 iscsi-ha: 12593 service_execute: Returning exit status [ 0 ]
Dec 16 09:33:18 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:18 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:18 xen-01 iscsi-ha: 12593 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431652 nr:3152 dw:4369708 dr:1325207948 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:33:18 xen-01 iscsi-ha: 12593 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 DRBD Resource: iscsi1 in Connected state
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 service_execute: systemctl mode being used
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 service_execute: Returning exit status [ 0 ]
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 CHECKING IP 127.0.0.1
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 CHECKING IP 172.16.14.199
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 CHECKING IP 172.16.14.47
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 CHECKING IP 10.10.10.1
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 CHECKING IP 10.10.10.3
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:33:19 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:33:19 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:33:19 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:33:19 xen-01 iscsi-ha: 12593 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:33:19 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:33:19 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:33:19 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:33:20 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 3 on PIDS: 9831
Dec 16 09:33:22 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:33:23 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:33:23 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:33:23 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:33:23 xen-01 ha-lizard:  TIMENOW = 1481880803
Dec 16 09:33:23 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:33:23 xen-01 ha-lizard:  TIMEELAPSED = 38
Dec 16 09:33:23 xen-01 ha-lizard:  MGT link failure duration = [ 38 ] Tolerance = [ 5 seconds ]
Dec 16 09:33:23 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:23 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:23 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:33:23 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:28 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:33:28 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:33:28 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:33:28 xen-01 ha-lizard:  TIMENOW = 1481880808
Dec 16 09:33:28 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:33:28 xen-01 ha-lizard:  TIMEELAPSED = 43
Dec 16 09:33:28 xen-01 ha-lizard:  MGT link failure duration = [ 43 ] Tolerance = [ 5 seconds ]
Dec 16 09:33:28 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:28 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:28 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:33:28 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:28 xen-01 iscsi-ha: 12579 Spawning new instance of iscsi-ha
Dec 16 09:33:28 xen-01 iscsi-ha: 12579 check_logger_processes Checking logger processes
Dec 16 09:33:28 xen-01 iscsi-ha: 12579 check_logger_processes No processes to clear
Dec 16 09:33:28 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:33:28 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:33:28 xen-01 iscsi-ha:  This iteration is count 6937
Dec 16 09:33:28 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:28 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:33:28 xen-01 iscsi-ha: 12890 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 service_execute: System V mode detected
Dec 16 09:33:29 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 service_execute: Returning exit status [ 0 ]
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431652 nr:3152 dw:4369708 dr:1325207964 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:33:29 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:29 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 DRBD Resource: iscsi1 in Connected state
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 service_execute: systemctl mode being used
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 service_execute: Returning exit status [ 0 ]
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 CHECKING IP 127.0.0.1
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 CHECKING IP 172.16.14.199
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 CHECKING IP 172.16.14.47
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 CHECKING IP 10.10.10.1
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 CHECKING IP 10.10.10.3
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:33:29 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:33:29 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:33:29 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:33:29 xen-01 iscsi-ha: 12890 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:33:29 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:33:29 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:33:29 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:33:30 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 4 on PIDS: 9831
Dec 16 09:33:32 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:33:33 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:33:33 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:33 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:33:33 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:33:33 xen-01 ha-lizard:  TIMENOW = 1481880813
Dec 16 09:33:33 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:33:33 xen-01 ha-lizard:  TIMEELAPSED = 48
Dec 16 09:33:33 xen-01 ha-lizard:  MGT link failure duration = [ 48 ] Tolerance = [ 5 seconds ]
Dec 16 09:33:33 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:33 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:33 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:33:38 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:38 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:33:38 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:33:38 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:33:38 xen-01 ha-lizard:  TIMENOW = 1481880818
Dec 16 09:33:38 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:33:38 xen-01 ha-lizard:  TIMEELAPSED = 53
Dec 16 09:33:38 xen-01 ha-lizard:  MGT link failure duration = [ 53 ] Tolerance = [ 5 seconds ]
Dec 16 09:33:38 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:38 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:38 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:33:39 xen-01 iscsi-ha: 12876 Spawning new instance of iscsi-ha
Dec 16 09:33:39 xen-01 iscsi-ha: 12876 check_logger_processes Checking logger processes
Dec 16 09:33:39 xen-01 iscsi-ha: 12876 check_logger_processes No processes to clear
Dec 16 09:33:39 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:33:39 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:33:39 xen-01 iscsi-ha:  This iteration is count 6938
Dec 16 09:33:39 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:39 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 service_execute: System V mode detected
Dec 16 09:33:39 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 service_execute: Returning exit status [ 0 ]
Dec 16 09:33:39 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:39 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431652 nr:3152 dw:4369708 dr:1325207980 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 DRBD Resource: iscsi1 in Connected state
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 service_execute: systemctl mode being used
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 service_execute: Returning exit status [ 0 ]
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 CHECKING IP 127.0.0.1
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 CHECKING IP 172.16.14.199
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 CHECKING IP 172.16.14.47
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 CHECKING IP 10.10.10.1
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 CHECKING IP 10.10.10.3
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:33:39 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:33:39 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:33:39 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:33:39 xen-01 iscsi-ha: 14181 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:33:39 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:33:39 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:33:39 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:33:40 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 5 on PIDS: 9831
Dec 16 09:33:42 xen-01 xsconsole: Exiting sleep
Dec 16 09:33:42 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:33:43 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:43 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:33:43 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:33:44 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:33:44 xen-01 ha-lizard:  TIMENOW = 1481880824
Dec 16 09:33:44 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:33:44 xen-01 ha-lizard:  TIMEELAPSED = 59
Dec 16 09:33:44 xen-01 ha-lizard:  MGT link failure duration = [ 59 ] Tolerance = [ 5 seconds ]
Dec 16 09:33:44 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:44 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:44 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:33:46 xen-01 xsconsole: User authenticated successfully
Dec 16 09:33:46 xen-01 xsconsole: Local shell
Dec 16 09:33:46 xen-01 xsconsole: Data update connection failed - retrying.  Exception was:
Dec 16 09:33:46 xen-01 xsconsole: ['HANDLE_INVALID', 'session', 'OpaqueRef:3a5700da-db8f-795f-8dfe-b10fb7874944']
Dec 16 09:33:48 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:49 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:33:49 xen-01 iscsi-ha: 14162 Spawning new instance of iscsi-ha
Dec 16 09:33:49 xen-01 iscsi-ha: 14162 check_logger_processes Checking logger processes
Dec 16 09:33:49 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:33:49 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:33:49 xen-01 ha-lizard:  TIMENOW = 1481880829
Dec 16 09:33:49 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:33:49 xen-01 ha-lizard:  TIMEELAPSED = 64
Dec 16 09:33:49 xen-01 ha-lizard:  MGT link failure duration = [ 64 ] Tolerance = [ 5 seconds ]
Dec 16 09:33:49 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:49 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:49 xen-01 iscsi-ha: 14162 check_logger_processes No processes to clear
Dec 16 09:33:49 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:33:49 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:33:49 xen-01 iscsi-ha:  This iteration is count 6939
Dec 16 09:33:49 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:49 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 service_execute: System V mode detected
Dec 16 09:33:49 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:49 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 service_execute: Returning exit status [ 0 ]
Dec 16 09:33:49 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:49 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431652 nr:3152 dw:4369708 dr:1325207996 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 DRBD Resource: iscsi1 in Connected state
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 service_execute: systemctl mode being used
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 service_execute: Returning exit status [ 0 ]
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 CHECKING IP 127.0.0.1
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 CHECKING IP 172.16.14.199
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 CHECKING IP 172.16.14.47
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 CHECKING IP 10.10.10.1
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 CHECKING IP 10.10.10.3
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:33:49 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:33:49 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:33:49 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:33:49 xen-01 iscsi-ha: 14787 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:33:49 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:33:49 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:33:49 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:33:50 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 6 on PIDS: 9831
Dec 16 09:33:52 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:33:53 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:54 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:33:54 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:33:54 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:33:54 xen-01 ha-lizard:  TIMENOW = 1481880834
Dec 16 09:33:54 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:33:54 xen-01 ha-lizard:  TIMEELAPSED = 69
Dec 16 09:33:54 xen-01 ha-lizard:  MGT link failure duration = [ 69 ] Tolerance = [ 5 seconds ]
Dec 16 09:33:54 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:54 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:54 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:33:58 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:33:59 xen-01 iscsi-ha: 14736 Spawning new instance of iscsi-ha
Dec 16 09:33:59 xen-01 iscsi-ha: 14736 check_logger_processes Checking logger processes
Dec 16 09:33:59 xen-01 iscsi-ha: 14736 check_logger_processes No processes to clear
Dec 16 09:33:59 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:33:59 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:33:59 xen-01 iscsi-ha:  This iteration is count 6940
Dec 16 09:33:59 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:33:59 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 service_execute: System V mode detected
Dec 16 09:33:59 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:59 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 service_execute: Returning exit status [ 0 ]
Dec 16 09:33:59 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:59 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431656 nr:3152 dw:4369712 dr:1325208012 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 DRBD Resource: iscsi1 in Connected state
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 service_execute: systemctl mode being used
Dec 16 09:33:59 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:33:59 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 service_execute: Returning exit status [ 0 ]
Dec 16 09:33:59 xen-01 ha-lizard:  TIMENOW = 1481880839
Dec 16 09:33:59 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:33:59 xen-01 ha-lizard:  TIMEELAPSED = 74
Dec 16 09:33:59 xen-01 ha-lizard:  MGT link failure duration = [ 74 ] Tolerance = [ 5 seconds ]
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:33:59 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:59 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 CHECKING IP 127.0.0.1
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 CHECKING IP 172.16.14.199
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 CHECKING IP 172.16.14.47
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 CHECKING IP 10.10.10.1
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 CHECKING IP 10.10.10.3
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:33:59 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:33:59 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:33:59 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:33:59 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:33:59 xen-01 iscsi-ha: 15161 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:33:59 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:33:59 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:33:59 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:34:00 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 7 on PIDS: 9831
Dec 16 09:34:02 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:34:03 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:04 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:34:04 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:34:04 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:34:04 xen-01 ha-lizard:  TIMENOW = 1481880844
Dec 16 09:34:04 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:34:04 xen-01 ha-lizard:  TIMEELAPSED = 79
Dec 16 09:34:04 xen-01 ha-lizard:  MGT link failure duration = [ 79 ] Tolerance = [ 5 seconds ]
Dec 16 09:34:04 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:04 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:05 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:34:08 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:09 xen-01 iscsi-ha: 15147 Spawning new instance of iscsi-ha
Dec 16 09:34:09 xen-01 iscsi-ha: 15147 check_logger_processes Checking logger processes
Dec 16 09:34:09 xen-01 iscsi-ha: 15147 check_logger_processes No processes to clear
Dec 16 09:34:09 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:34:09 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:34:09 xen-01 iscsi-ha:  This iteration is count 6941
Dec 16 09:34:09 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:09 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 service_execute: System V mode detected
Dec 16 09:34:09 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 service_execute: Returning exit status [ 0 ]
Dec 16 09:34:09 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:09 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431656 nr:3152 dw:4369712 dr:1325208028 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 DRBD Resource: iscsi1 in Connected state
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 service_execute: systemctl mode being used
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 service_execute: Returning exit status [ 0 ]
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 CHECKING IP 127.0.0.1
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 CHECKING IP 172.16.14.199
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 CHECKING IP 172.16.14.47
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 CHECKING IP 10.10.10.1
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 CHECKING IP 10.10.10.3
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:34:09 xen-01 iscsi-ha: 15742 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:34:10 xen-01 iscsi-ha: 15742 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:34:10 xen-01 iscsi-ha: 15742 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:34:10 xen-01 iscsi-ha: 15742 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:34:10 xen-01 iscsi-ha: 15742 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:34:10 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:34:10 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:34:10 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:34:10 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:34:10 xen-01 iscsi-ha: 15742 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:34:10 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:34:10 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:34:10 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:34:10 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:34:10 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:34:10 xen-01 ha-lizard:  TIMENOW = 1481880850
Dec 16 09:34:10 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:34:10 xen-01 ha-lizard:  TIMEELAPSED = 85
Dec 16 09:34:10 xen-01 ha-lizard:  MGT link failure duration = [ 85 ] Tolerance = [ 5 seconds ]
Dec 16 09:34:10 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:10 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:10 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:34:10 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 8 on PIDS: 9831
Dec 16 09:34:12 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:34:13 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:15 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:34:15 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:34:15 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:34:15 xen-01 ha-lizard:  TIMENOW = 1481880855
Dec 16 09:34:15 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:34:15 xen-01 ha-lizard:  TIMEELAPSED = 90
Dec 16 09:34:15 xen-01 ha-lizard:  MGT link failure duration = [ 90 ] Tolerance = [ 5 seconds ]
Dec 16 09:34:15 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:15 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:15 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:34:18 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:19 xen-01 iscsi-ha: 15719 Spawning new instance of iscsi-ha
Dec 16 09:34:19 xen-01 iscsi-ha: 15719 check_logger_processes Checking logger processes
Dec 16 09:34:19 xen-01 iscsi-ha: 15719 check_logger_processes No processes to clear
Dec 16 09:34:19 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:34:19 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:34:19 xen-01 iscsi-ha:  This iteration is count 6942
Dec 16 09:34:19 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:19 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:34:19 xen-01 iscsi-ha: 16835 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:34:19 xen-01 iscsi-ha: 16835 service_execute: System V mode detected
Dec 16 09:34:19 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:19 xen-01 iscsi-ha: 16835 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:34:19 xen-01 iscsi-ha: 16835 service_execute: Returning exit status [ 0 ]
Dec 16 09:34:19 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:19 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:19 xen-01 iscsi-ha: 16835 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431656 nr:3152 dw:4369712 dr:1325208044 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 DRBD Resource: iscsi1 in Connected state
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 service_execute: systemctl mode being used
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 service_execute: Returning exit status [ 0 ]
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 CHECKING IP 127.0.0.1
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 CHECKING IP 172.16.14.199
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 CHECKING IP 172.16.14.47
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 CHECKING IP 10.10.10.1
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 CHECKING IP 10.10.10.3
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:34:20 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:34:20 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:34:20 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:34:20 xen-01 iscsi-ha: 16835 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:34:20 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:34:20 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:34:20 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:34:20 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:34:20 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:34:20 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:34:20 xen-01 ha-lizard:  TIMENOW = 1481880860
Dec 16 09:34:20 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:34:20 xen-01 ha-lizard:  TIMEELAPSED = 95
Dec 16 09:34:20 xen-01 ha-lizard:  MGT link failure duration = [ 95 ] Tolerance = [ 5 seconds ]
Dec 16 09:34:20 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:20 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:20 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:34:20 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 9 on PIDS: 9831
Dec 16 09:34:22 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:34:23 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:25 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:34:25 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:34:25 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:34:25 xen-01 ha-lizard:  TIMENOW = 1481880865
Dec 16 09:34:26 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:34:26 xen-01 ha-lizard:  TIMEELAPSED = 100
Dec 16 09:34:26 xen-01 ha-lizard:  MGT link failure duration = [ 100 ] Tolerance = [ 5 seconds ]
Dec 16 09:34:26 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:26 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:26 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:34:28 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:29 xen-01 iscsi-ha: 16801 Spawning new instance of iscsi-ha
Dec 16 09:34:29 xen-01 iscsi-ha: 16801 check_logger_processes Checking logger processes
Dec 16 09:34:29 xen-01 iscsi-ha: 16801 check_logger_processes No processes to clear
Dec 16 09:34:29 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:34:29 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:34:30 xen-01 iscsi-ha:  This iteration is count 6943
Dec 16 09:34:30 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:30 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 service_execute: System V mode detected
Dec 16 09:34:30 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 service_execute: Returning exit status [ 0 ]
Dec 16 09:34:30 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:30 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431664 nr:3152 dw:4369720 dr:1325208060 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 DRBD Resource: iscsi1 in Connected state
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 service_execute: systemctl mode being used
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 service_execute: Returning exit status [ 0 ]
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 CHECKING IP 127.0.0.1
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 CHECKING IP 172.16.14.199
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 CHECKING IP 172.16.14.47
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 CHECKING IP 10.10.10.1
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 CHECKING IP 10.10.10.3
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:34:30 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:34:30 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:34:30 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:34:30 xen-01 iscsi-ha: 17310 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:34:30 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:34:30 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:34:30 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:34:30 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 10 on PIDS: 9831
Dec 16 09:34:31 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:34:31 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:34:31 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:34:31 xen-01 ha-lizard:  TIMENOW = 1481880871
Dec 16 09:34:31 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:34:31 xen-01 ha-lizard:  TIMEELAPSED = 106
Dec 16 09:34:31 xen-01 ha-lizard:  MGT link failure duration = [ 106 ] Tolerance = [ 5 seconds ]
Dec 16 09:34:31 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:31 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:31 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:34:32 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:34:33 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:36 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:34:36 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:34:36 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:34:37 xen-01 ha-lizard:  TIMENOW = 1481880877
Dec 16 09:34:37 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:34:37 xen-01 ha-lizard:  TIMEELAPSED = 112
Dec 16 09:34:37 xen-01 ha-lizard:  MGT link failure duration = [ 112 ] Tolerance = [ 5 seconds ]
Dec 16 09:34:37 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:37 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:37 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:34:38 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:40 xen-01 iscsi-ha: 17296 Spawning new instance of iscsi-ha
Dec 16 09:34:40 xen-01 iscsi-ha: 17296 check_logger_processes Checking logger processes
Dec 16 09:34:40 xen-01 iscsi-ha: 17296 check_logger_processes No processes to clear
Dec 16 09:34:40 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:34:40 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:34:40 xen-01 iscsi-ha:  This iteration is count 6944
Dec 16 09:34:40 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:40 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 service_execute: System V mode detected
Dec 16 09:34:40 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 service_execute: Returning exit status [ 0 ]
Dec 16 09:34:40 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:40 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431664 nr:3152 dw:4369720 dr:1325208076 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 DRBD Resource: iscsi1 in Connected state
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 service_execute: systemctl mode being used
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 service_execute: Returning exit status [ 0 ]
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 CHECKING IP 127.0.0.1
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 CHECKING IP 172.16.14.199
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 CHECKING IP 172.16.14.47
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 CHECKING IP 10.10.10.1
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 CHECKING IP 10.10.10.3
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:34:40 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:34:40 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:34:40 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:34:40 xen-01 iscsi-ha: 17681 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:34:40 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:34:40 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:34:40 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:34:41 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 11 on PIDS: 9831
Dec 16 09:34:42 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:34:42 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:34:42 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:34:42 xen-01 ha-lizard:  TIMENOW = 1481880882
Dec 16 09:34:42 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:34:42 xen-01 ha-lizard:  TIMEELAPSED = 117
Dec 16 09:34:42 xen-01 ha-lizard:  MGT link failure duration = [ 117 ] Tolerance = [ 5 seconds ]
Dec 16 09:34:42 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:42 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:42 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:34:42 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:34:43 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:47 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:34:47 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:34:47 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:34:47 xen-01 ha-lizard:  TIMENOW = 1481880887
Dec 16 09:34:47 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:34:47 xen-01 ha-lizard:  TIMEELAPSED = 122
Dec 16 09:34:47 xen-01 ha-lizard:  MGT link failure duration = [ 122 ] Tolerance = [ 5 seconds ]
Dec 16 09:34:47 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:47 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:48 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:34:48 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:50 xen-01 iscsi-ha: 17667 Spawning new instance of iscsi-ha
Dec 16 09:34:50 xen-01 iscsi-ha: 17667 check_logger_processes Checking logger processes
Dec 16 09:34:50 xen-01 iscsi-ha: 17667 check_logger_processes No processes to clear
Dec 16 09:34:50 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:34:50 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:34:50 xen-01 iscsi-ha:  This iteration is count 6945
Dec 16 09:34:50 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:34:50 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 service_execute: System V mode detected
Dec 16 09:34:50 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 service_execute: Returning exit status [ 0 ]
Dec 16 09:34:50 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:50 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431664 nr:3152 dw:4369720 dr:1325208092 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 DRBD Resource: iscsi1 in Connected state
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 service_execute: systemctl mode being used
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 service_execute: Returning exit status [ 0 ]
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 CHECKING IP 127.0.0.1
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 CHECKING IP 172.16.14.199
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 CHECKING IP 172.16.14.47
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 CHECKING IP 10.10.10.1
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 CHECKING IP 10.10.10.3
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:34:50 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:34:50 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:34:50 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:34:50 xen-01 iscsi-ha: 18943 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:34:50 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:34:50 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:34:50 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:34:51 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 12 on PIDS: 9831
Dec 16 09:34:52 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:34:53 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:34:53 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:34:53 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:34:53 xen-01 ha-lizard:  TIMENOW = 1481880893
Dec 16 09:34:53 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:34:53 xen-01 ha-lizard:  TIMEELAPSED = 128
Dec 16 09:34:53 xen-01 ha-lizard:  MGT link failure duration = [ 128 ] Tolerance = [ 5 seconds ]
Dec 16 09:34:53 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:53 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:53 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:34:53 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:58 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:34:58 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:34:58 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:34:58 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:34:58 xen-01 ha-lizard:  TIMENOW = 1481880898
Dec 16 09:34:58 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:34:58 xen-01 ha-lizard:  TIMEELAPSED = 133
Dec 16 09:34:58 xen-01 ha-lizard:  MGT link failure duration = [ 133 ] Tolerance = [ 5 seconds ]
Dec 16 09:34:58 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:58 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:34:59 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:35:00 xen-01 iscsi-ha: 18929 Spawning new instance of iscsi-ha
Dec 16 09:35:00 xen-01 iscsi-ha: 18929 check_logger_processes Checking logger processes
Dec 16 09:35:00 xen-01 iscsi-ha: 18929 check_logger_processes No processes to clear
Dec 16 09:35:00 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:35:00 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:35:00 xen-01 iscsi-ha:  This iteration is count 6946
Dec 16 09:35:00 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:00 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 service_execute: System V mode detected
Dec 16 09:35:00 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 service_execute: Returning exit status [ 0 ]
Dec 16 09:35:00 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:00 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431668 nr:3152 dw:4369724 dr:1325208108 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 DRBD Resource: iscsi1 in Connected state
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 service_execute: systemctl mode being used
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 service_execute: Returning exit status [ 0 ]
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 CHECKING IP 127.0.0.1
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 CHECKING IP 172.16.14.199
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 CHECKING IP 172.16.14.47
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 CHECKING IP 10.10.10.1
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 CHECKING IP 10.10.10.3
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:35:00 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:35:00 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:35:00 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:35:00 xen-01 iscsi-ha: 19428 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:35:00 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:35:00 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:35:00 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:35:01 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 13 on PIDS: 9831
Dec 16 09:35:02 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:35:03 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:04 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:35:04 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:35:04 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:35:04 xen-01 ha-lizard:  TIMENOW = 1481880904
Dec 16 09:35:04 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:35:04 xen-01 ha-lizard:  TIMEELAPSED = 139
Dec 16 09:35:04 xen-01 ha-lizard:  MGT link failure duration = [ 139 ] Tolerance = [ 5 seconds ]
Dec 16 09:35:04 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:04 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:04 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:35:08 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:09 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:35:09 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:35:09 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:35:09 xen-01 ha-lizard:  TIMENOW = 1481880909
Dec 16 09:35:09 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:35:09 xen-01 ha-lizard:  TIMEELAPSED = 144
Dec 16 09:35:09 xen-01 ha-lizard:  MGT link failure duration = [ 144 ] Tolerance = [ 5 seconds ]
Dec 16 09:35:10 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:10 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:10 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:35:10 xen-01 iscsi-ha: 19414 Spawning new instance of iscsi-ha
Dec 16 09:35:10 xen-01 iscsi-ha: 19414 check_logger_processes Checking logger processes
Dec 16 09:35:10 xen-01 iscsi-ha: 19414 check_logger_processes No processes to clear
Dec 16 09:35:10 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:35:10 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:35:10 xen-01 iscsi-ha:  This iteration is count 6947
Dec 16 09:35:10 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:10 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 service_execute: System V mode detected
Dec 16 09:35:10 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 service_execute: Returning exit status [ 0 ]
Dec 16 09:35:10 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:10 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431668 nr:3152 dw:4369724 dr:1325208124 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 DRBD Resource: iscsi1 in Connected state
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 service_execute: systemctl mode being used
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 service_execute: Returning exit status [ 0 ]
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 CHECKING IP 127.0.0.1
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 CHECKING IP 172.16.14.199
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 CHECKING IP 172.16.14.47
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 CHECKING IP 10.10.10.1
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 CHECKING IP 10.10.10.3
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:35:10 xen-01 iscsi-ha: 19794 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:35:11 xen-01 iscsi-ha: 19794 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:35:11 xen-01 iscsi-ha: 19794 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:35:11 xen-01 iscsi-ha: 19794 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:35:11 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:35:11 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:35:11 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:35:11 xen-01 iscsi-ha: 19794 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:35:11 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:35:11 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:35:11 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:35:11 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 14 on PIDS: 9831
Dec 16 09:35:12 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:35:13 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:15 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:35:15 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:35:15 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:35:15 xen-01 ha-lizard:  TIMENOW = 1481880915
Dec 16 09:35:15 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:35:15 xen-01 ha-lizard:  TIMEELAPSED = 150
Dec 16 09:35:15 xen-01 ha-lizard:  MGT link failure duration = [ 150 ] Tolerance = [ 5 seconds ]
Dec 16 09:35:15 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:15 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:15 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:35:18 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:20 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:35:20 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:35:20 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:35:20 xen-01 ha-lizard:  TIMENOW = 1481880920
Dec 16 09:35:20 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:35:20 xen-01 ha-lizard:  TIMEELAPSED = 155
Dec 16 09:35:20 xen-01 ha-lizard:  MGT link failure duration = [ 155 ] Tolerance = [ 5 seconds ]
Dec 16 09:35:20 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:20 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:20 xen-01 iscsi-ha: 19780 Spawning new instance of iscsi-ha
Dec 16 09:35:20 xen-01 iscsi-ha: 19780 check_logger_processes Checking logger processes
Dec 16 09:35:20 xen-01 iscsi-ha: 19780 check_logger_processes No processes to clear
Dec 16 09:35:20 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:35:20 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:35:20 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:35:20 xen-01 iscsi-ha:  This iteration is count 6948
Dec 16 09:35:20 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:20 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:35:20 xen-01 iscsi-ha: 20722 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:35:20 xen-01 iscsi-ha: 20722 service_execute: System V mode detected
Dec 16 09:35:20 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 service_execute: Returning exit status [ 0 ]
Dec 16 09:35:21 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:21 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431668 nr:3152 dw:4369724 dr:1325208140 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 DRBD Resource: iscsi1 in Connected state
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 service_execute: systemctl mode being used
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 service_execute: Returning exit status [ 0 ]
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 CHECKING IP 127.0.0.1
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 CHECKING IP 172.16.14.199
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 CHECKING IP 172.16.14.47
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 CHECKING IP 10.10.10.1
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 CHECKING IP 10.10.10.3
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:35:21 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:35:21 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:35:21 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:35:21 xen-01 iscsi-ha: 20722 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:35:21 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:35:21 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:35:21 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:35:21 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 15 on PIDS: 9831
Dec 16 09:35:22 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:35:23 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:25 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:35:26 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:35:26 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:35:26 xen-01 ha-lizard:  TIMENOW = 1481880926
Dec 16 09:35:26 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:35:26 xen-01 ha-lizard:  TIMEELAPSED = 161
Dec 16 09:35:26 xen-01 ha-lizard:  MGT link failure duration = [ 161 ] Tolerance = [ 5 seconds ]
Dec 16 09:35:26 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:26 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:26 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:35:28 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:30 xen-01 iscsi-ha: 20696 Spawning new instance of iscsi-ha
Dec 16 09:35:30 xen-01 iscsi-ha: 20696 check_logger_processes Checking logger processes
Dec 16 09:35:31 xen-01 iscsi-ha: 20696 check_logger_processes No processes to clear
Dec 16 09:35:31 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:35:31 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:35:31 xen-01 iscsi-ha:  This iteration is count 6949
Dec 16 09:35:31 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:31 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 service_execute: System V mode detected
Dec 16 09:35:31 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 service_execute: Returning exit status [ 0 ]
Dec 16 09:35:31 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:31 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431684 nr:3152 dw:4369740 dr:1325208156 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 DRBD Resource: iscsi1 in Connected state
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 service_execute: systemctl mode being used
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 service_execute: Returning exit status [ 0 ]
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 CHECKING IP 127.0.0.1
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 CHECKING IP 172.16.14.199
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 CHECKING IP 172.16.14.47
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 CHECKING IP 10.10.10.1
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 CHECKING IP 10.10.10.3
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:35:31 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:35:31 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:35:31 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:35:31 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:35:31 xen-01 iscsi-ha: 21487 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:35:31 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:35:31 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:35:31 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:35:31 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:35:31 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:35:31 xen-01 ha-lizard:  TIMENOW = 1481880931
Dec 16 09:35:31 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:35:31 xen-01 ha-lizard:  TIMEELAPSED = 166
Dec 16 09:35:31 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 16 on PIDS: 9831
Dec 16 09:35:31 xen-01 ha-lizard:  MGT link failure duration = [ 166 ] Tolerance = [ 5 seconds ]
Dec 16 09:35:31 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:31 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:31 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:35:32 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:35:33 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:36 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:35:37 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:35:37 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:35:37 xen-01 ha-lizard:  TIMENOW = 1481880937
Dec 16 09:35:37 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:35:37 xen-01 ha-lizard:  TIMEELAPSED = 172
Dec 16 09:35:37 xen-01 ha-lizard:  MGT link failure duration = [ 172 ] Tolerance = [ 5 seconds ]
Dec 16 09:35:37 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:37 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:37 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:35:38 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:41 xen-01 iscsi-ha: 21473 Spawning new instance of iscsi-ha
Dec 16 09:35:41 xen-01 iscsi-ha: 21473 check_logger_processes Checking logger processes
Dec 16 09:35:41 xen-01 iscsi-ha: 21473 check_logger_processes No processes to clear
Dec 16 09:35:41 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:35:41 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:35:41 xen-01 iscsi-ha:  This iteration is count 6950
Dec 16 09:35:41 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:41 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 service_execute: System V mode detected
Dec 16 09:35:41 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 service_execute: Returning exit status [ 0 ]
Dec 16 09:35:41 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:41 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431684 nr:3152 dw:4369740 dr:1325208172 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 DRBD Resource: iscsi1 in Connected state
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 service_execute: systemctl mode being used
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 service_execute: Returning exit status [ 0 ]
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 CHECKING IP 127.0.0.1
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 CHECKING IP 172.16.14.199
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 CHECKING IP 172.16.14.47
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 CHECKING IP 10.10.10.1
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 CHECKING IP 10.10.10.3
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:35:41 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:35:41 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:35:41 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:35:41 xen-01 iscsi-ha: 21851 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:35:41 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:35:41 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:35:41 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:35:41 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 17 on PIDS: 9831
Dec 16 09:35:42 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:35:42 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:35:42 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:35:42 xen-01 ha-lizard:  TIMENOW = 1481880942
Dec 16 09:35:42 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:35:42 xen-01 ha-lizard:  TIMEELAPSED = 177
Dec 16 09:35:42 xen-01 ha-lizard:  MGT link failure duration = [ 177 ] Tolerance = [ 5 seconds ]
Dec 16 09:35:42 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:42 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:42 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:35:42 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:35:43 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:47 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:35:48 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:35:48 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:35:48 xen-01 ha-lizard:  TIMENOW = 1481880948
Dec 16 09:35:48 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:35:48 xen-01 ha-lizard:  TIMEELAPSED = 183
Dec 16 09:35:48 xen-01 ha-lizard:  MGT link failure duration = [ 183 ] Tolerance = [ 5 seconds ]
Dec 16 09:35:48 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:48 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:48 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:35:48 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:51 xen-01 iscsi-ha: 21837 Spawning new instance of iscsi-ha
Dec 16 09:35:51 xen-01 iscsi-ha: 21837 check_logger_processes Checking logger processes
Dec 16 09:35:51 xen-01 iscsi-ha: 21837 check_logger_processes No processes to clear
Dec 16 09:35:51 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:35:51 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:35:51 xen-01 iscsi-ha:  This iteration is count 6951
Dec 16 09:35:51 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:35:51 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 service_execute: System V mode detected
Dec 16 09:35:51 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 service_execute: Returning exit status [ 0 ]
Dec 16 09:35:51 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:51 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431684 nr:3152 dw:4369740 dr:1325208188 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 DRBD Resource: iscsi1 in Connected state
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 service_execute: systemctl mode being used
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 service_execute: Returning exit status [ 0 ]
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 CHECKING IP 127.0.0.1
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 CHECKING IP 172.16.14.199
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 CHECKING IP 172.16.14.47
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 CHECKING IP 10.10.10.1
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 CHECKING IP 10.10.10.3
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:35:51 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:35:51 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:35:51 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:35:51 xen-01 iscsi-ha: 22220 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:35:51 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:35:51 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:35:51 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:35:51 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 18 on PIDS: 9831
Dec 16 09:35:52 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:35:53 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:35:53 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:35:53 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:35:53 xen-01 ha-lizard:  TIMENOW = 1481880953
Dec 16 09:35:53 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:35:53 xen-01 ha-lizard:  TIMEELAPSED = 188
Dec 16 09:35:53 xen-01 ha-lizard:  MGT link failure duration = [ 188 ] Tolerance = [ 5 seconds ]
Dec 16 09:35:53 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:53 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:53 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:54 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:35:58 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:35:59 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:35:59 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:35:59 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:35:59 xen-01 ha-lizard:  TIMENOW = 1481880959
Dec 16 09:35:59 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:35:59 xen-01 ha-lizard:  TIMEELAPSED = 194
Dec 16 09:35:59 xen-01 ha-lizard:  MGT link failure duration = [ 194 ] Tolerance = [ 5 seconds ]
Dec 16 09:35:59 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:59 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:35:59 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:36:01 xen-01 iscsi-ha: 22206 Spawning new instance of iscsi-ha
Dec 16 09:36:01 xen-01 iscsi-ha: 22206 check_logger_processes Checking logger processes
Dec 16 09:36:01 xen-01 iscsi-ha: 22206 check_logger_processes No processes to clear
Dec 16 09:36:01 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:36:01 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:36:01 xen-01 iscsi-ha:  This iteration is count 6952
Dec 16 09:36:01 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:01 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 service_execute: System V mode detected
Dec 16 09:36:01 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 service_execute: Returning exit status [ 0 ]
Dec 16 09:36:01 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:01 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431688 nr:3152 dw:4369744 dr:1325208204 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 DRBD Resource: iscsi1 in Connected state
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 service_execute: systemctl mode being used
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 service_execute: Returning exit status [ 0 ]
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 CHECKING IP 127.0.0.1
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 CHECKING IP 172.16.14.199
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 CHECKING IP 172.16.14.47
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 CHECKING IP 10.10.10.1
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 CHECKING IP 10.10.10.3
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:36:01 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 19 on PIDS: 9831
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:36:01 xen-01 iscsi-ha: 23601 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:36:01 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:36:01 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:36:01 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:36:02 xen-01 iscsi-ha: 23601 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:36:02 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:36:02 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:36:02 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:36:02 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:36:03 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:04 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:36:04 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:36:04 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:36:04 xen-01 ha-lizard:  TIMENOW = 1481880964
Dec 16 09:36:04 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:36:04 xen-01 ha-lizard:  TIMEELAPSED = 199
Dec 16 09:36:04 xen-01 ha-lizard:  MGT link failure duration = [ 199 ] Tolerance = [ 5 seconds ]
Dec 16 09:36:04 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:04 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:05 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:36:08 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:10 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:36:10 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:36:10 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:36:10 xen-01 ha-lizard:  TIMENOW = 1481880970
Dec 16 09:36:10 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:36:10 xen-01 ha-lizard:  TIMEELAPSED = 205
Dec 16 09:36:10 xen-01 ha-lizard:  MGT link failure duration = [ 205 ] Tolerance = [ 5 seconds ]
Dec 16 09:36:10 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:10 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:10 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:36:11 xen-01 iscsi-ha: 23587 Spawning new instance of iscsi-ha
Dec 16 09:36:11 xen-01 iscsi-ha: 23587 check_logger_processes Checking logger processes
Dec 16 09:36:11 xen-01 iscsi-ha: 23587 check_logger_processes No processes to clear
Dec 16 09:36:11 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:36:11 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:36:11 xen-01 iscsi-ha:  This iteration is count 6953
Dec 16 09:36:11 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:11 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:36:11 xen-01 iscsi-ha: 23959 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:36:11 xen-01 iscsi-ha: 23959 service_execute: System V mode detected
Dec 16 09:36:11 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:11 xen-01 iscsi-ha: 23959 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:36:11 xen-01 iscsi-ha: 23959 service_execute: Returning exit status [ 0 ]
Dec 16 09:36:11 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:11 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:11 xen-01 iscsi-ha: 23959 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431688 nr:3152 dw:4369744 dr:1325208220 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:36:11 xen-01 iscsi-ha: 23959 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:36:11 xen-01 iscsi-ha: 23959 DRBD Resource: iscsi1 in Connected state
Dec 16 09:36:11 xen-01 iscsi-ha: 23959 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:36:11 xen-01 iscsi-ha: 23959 service_execute: systemctl mode being used
Dec 16 09:36:11 xen-01 iscsi-ha: 23959 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:36:11 xen-01 iscsi-ha: 23959 service_execute: Returning exit status [ 0 ]
Dec 16 09:36:11 xen-01 iscsi-ha: 23959 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:36:12 xen-01 iscsi-ha: 23959 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:36:12 xen-01 iscsi-ha: 23959 CHECKING IP 127.0.0.1
Dec 16 09:36:12 xen-01 iscsi-ha: 23959 CHECKING IP 172.16.14.199
Dec 16 09:36:12 xen-01 iscsi-ha: 23959 CHECKING IP 172.16.14.47
Dec 16 09:36:12 xen-01 iscsi-ha: 23959 CHECKING IP 10.10.10.1
Dec 16 09:36:12 xen-01 iscsi-ha: 23959 CHECKING IP 10.10.10.3
Dec 16 09:36:12 xen-01 iscsi-ha: 23959 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:36:12 xen-01 iscsi-ha: 23959 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:36:12 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 20 on PIDS: 9831
Dec 16 09:36:12 xen-01 iscsi-ha: 23959 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:36:12 xen-01 iscsi-ha: 23959 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:36:12 xen-01 iscsi-ha: 23959 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:36:12 xen-01 iscsi-ha: 23959 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:36:12 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:36:12 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:36:12 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:36:12 xen-01 iscsi-ha: 23959 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:36:12 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:36:12 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:36:12 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:36:12 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:36:13 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:15 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:36:15 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:36:15 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:36:15 xen-01 ha-lizard:  TIMENOW = 1481880975
Dec 16 09:36:15 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:36:15 xen-01 ha-lizard:  TIMEELAPSED = 210
Dec 16 09:36:15 xen-01 ha-lizard:  MGT link failure duration = [ 210 ] Tolerance = [ 5 seconds ]
Dec 16 09:36:15 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:15 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:15 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:36:18 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:20 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:36:20 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:36:20 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:36:20 xen-01 ha-lizard:  TIMENOW = 1481880980
Dec 16 09:36:20 xen-01 ha-lizard:  TIMEFAILED = 1481880765
Dec 16 09:36:20 xen-01 ha-lizard:  TIMEELAPSED = 215
Dec 16 09:36:20 xen-01 ha-lizard:  MGT link failure duration = [ 215 ] Tolerance = [ 5 seconds ]
Dec 16 09:36:20 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:20 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:20 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:36:21 xen-01 iscsi-ha: 23945 Spawning new instance of iscsi-ha
Dec 16 09:36:21 xen-01 iscsi-ha: 23945 check_logger_processes Checking logger processes
Dec 16 09:36:21 xen-01 iscsi-ha: 23945 check_logger_processes No processes to clear
Dec 16 09:36:21 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:36:21 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:36:21 xen-01 iscsi-ha:  This iteration is count 6954
Dec 16 09:36:21 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:21 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:36:21 xen-01 iscsi-ha: 24333 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:36:21 xen-01 iscsi-ha: 24333 service_execute: System V mode detected
Dec 16 09:36:21 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 service_execute: Returning exit status [ 0 ]
Dec 16 09:36:22 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:22 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431688 nr:3152 dw:4369744 dr:1325208236 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 DRBD Resource: iscsi1 in Connected state
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 service_execute: systemctl mode being used
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 service_execute: Returning exit status [ 0 ]
Dec 16 09:36:22 xen-01 ha-lizard: 9831 ha-lizard already running: Attempt 21 on PIDS: 9831
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:36:22 xen-01 ha-lizard: 9831 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 CHECKING IP 127.0.0.1
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 CHECKING IP 172.16.14.199
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 CHECKING IP 172.16.14.47
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 CHECKING IP 10.10.10.1
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 CHECKING IP 10.10.10.3
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:36:22 xen-01 ha-lizard: 9831 email: Sending ALERT email to root@localhost: ha-lizard failed to spawn new instance after 21 attmepts. MAX_STARTS is set to 20. Check Host: xen-01 for possible hung process
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:36:22 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:36:22 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:36:22 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email from: root@localhost
Dec 16 09:36:22 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Traceback (most recent call last):
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email to: root@localhost
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xen-01
Dec 16 09:36:22 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/etc/ha-lizard/scripts/email_alert.py", line 105, in <module>
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Timestamp: Fri Dec 16 09:36:22 GMT 2016
Dec 16 09:36:22 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     message = smtplib.SMTP(smtp_server, smtp_port, hostname)
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Process: ha-lizard Version: 2.1.3
Dec 16 09:36:22 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Content: ha-lizard failed to spawn new instance after 21 attmepts. MAX_STARTS is set to 20. Check Host: xen-01 for possible hung process
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: 
Dec 16 09:36:22 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     (code, msg) = self.connect(host, port)
Dec 16 09:36:22 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 315, in connect
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool UUID       : e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Name-Label : XenPool
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master UUID: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:36:22 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     self.sock = self._get_socket(host, port, self.timeout)
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-01
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:36:22 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : true
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:36:22 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     return socket.create_connection((host, port), timeout)
Dec 16 09:36:22 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-02
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : false
Dec 16 09:36:22 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Hostname: xen-01
Dec 16 09:36:22 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: socket.error: [Errno 111] Connection refused
Dec 16 09:36:22 xen-01 ha-lizard: 9831 email: Message copied to /dev/shm/ha-lizard-mail/1481880982.msg - Suppressing duplicates for 60 Minutes
Dec 16 09:36:22 xen-01 ha-lizard: 9831 ha-lizard failed to spawn new instance after 21 attmepts. Check xen-01 for possible hung process
Dec 16 09:36:22 xen-01 ha-lizard: 9831 MONITOR has reached configured threshhold - 20 - MONITOR_KILLALL is enabled. - attempting to kill all existing processes
Dec 16 09:36:22 xen-01 iscsi-ha: 24333 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:36:22 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:36:22 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:36:22 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:36:22 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: /etc/ha-lizard/init/ha-lizard.mon: line 80:  9831 Terminated              $EXECUTE
Dec 16 09:36:22 xen-01 ha-lizard: 9831 Successfully killed all instances of ha-lizard.sh
Dec 16 09:36:22 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:36:23 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:28 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:31 xen-01 iscsi-ha: 24319 Spawning new instance of iscsi-ha
Dec 16 09:36:31 xen-01 iscsi-ha: 24319 check_logger_processes Checking logger processes
Dec 16 09:36:32 xen-01 iscsi-ha: 24319 check_logger_processes No processes to clear
Dec 16 09:36:32 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:36:32 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:36:32 xen-01 iscsi-ha:  This iteration is count 6955
Dec 16 09:36:32 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:32 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 service_execute: System V mode detected
Dec 16 09:36:32 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 service_execute: Returning exit status [ 0 ]
Dec 16 09:36:32 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:32 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431696 nr:3152 dw:4369752 dr:1325208252 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 DRBD Resource: iscsi1 in Connected state
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 service_execute: systemctl mode being used
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 service_execute: Returning exit status [ 0 ]
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 CHECKING IP 127.0.0.1
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 CHECKING IP 172.16.14.199
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 CHECKING IP 172.16.14.47
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 CHECKING IP 10.10.10.1
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 CHECKING IP 10.10.10.3
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:36:32 xen-01 ha-lizard: 9831 Spawning new instance of ha-lizard
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:36:32 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:36:32 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:36:32 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:36:32 xen-01 iscsi-ha: 25627 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:36:32 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:36:32 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:36:32 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:36:32 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:36:32 xen-01 ha-lizard:  This iteration is count 4630
Dec 16 09:36:32 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:32 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:36:32 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:36:32 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:36:32 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:36:32 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:36:32 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:36:32 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:36:32 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:36:32 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:36:32 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:36:32 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:36:33 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:36:33 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:36:33 xen-01 ha-lizard:  Master management link = DOWN
Dec 16 09:36:33 xen-01 ha-lizard:  TIMENOW = 1481880993
Dec 16 09:36:33 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:36:33 xen-01 ha-lizard:  TIMEELAPSED = 0
Dec 16 09:36:33 xen-01 ha-lizard:  MGT link failure duration = [ 0 ] Tolerance = [ 5 seconds ]
Dec 16 09:36:33 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:36:33 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:38 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:36:38 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:36:38 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:36:38 xen-01 ha-lizard:  TIMENOW = 1481880998
Dec 16 09:36:38 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:36:38 xen-01 ha-lizard:  TIMEELAPSED = 5
Dec 16 09:36:38 xen-01 ha-lizard:  MGT link failure duration = [ 5 ] Tolerance = [ 5 seconds ]
Dec 16 09:36:38 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:36:38 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:42 xen-01 iscsi-ha: 25606 Spawning new instance of iscsi-ha
Dec 16 09:36:42 xen-01 iscsi-ha: 25606 check_logger_processes Checking logger processes
Dec 16 09:36:42 xen-01 iscsi-ha: 25606 check_logger_processes No processes to clear
Dec 16 09:36:42 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:36:42 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:36:42 xen-01 iscsi-ha:  This iteration is count 6956
Dec 16 09:36:42 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:42 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 service_execute: System V mode detected
Dec 16 09:36:42 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 service_execute: Returning exit status [ 0 ]
Dec 16 09:36:42 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:42 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431696 nr:3152 dw:4369752 dr:1325208268 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 DRBD Resource: iscsi1 in Connected state
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 service_execute: systemctl mode being used
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 service_execute: Returning exit status [ 0 ]
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 CHECKING IP 127.0.0.1
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 CHECKING IP 172.16.14.199
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 CHECKING IP 172.16.14.47
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 CHECKING IP 10.10.10.1
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 CHECKING IP 10.10.10.3
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:36:42 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:36:42 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:36:42 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:36:42 xen-01 iscsi-ha: 26083 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:36:42 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:36:42 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:36:42 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:36:42 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:36:43 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:36:43 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:36:43 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:36:43 xen-01 ha-lizard:  TIMENOW = 1481881003
Dec 16 09:36:43 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:36:43 xen-01 ha-lizard:  TIMEELAPSED = 10
Dec 16 09:36:43 xen-01 ha-lizard:  MGT link failure duration = [ 10 ] Tolerance = [ 5 seconds ]
Dec 16 09:36:43 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:43 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:43 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:36:44 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:47 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 1 on PIDS: 25761
Dec 16 09:36:48 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:36:48 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:36:48 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:36:48 xen-01 ha-lizard:  TIMENOW = 1481881008
Dec 16 09:36:48 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:36:48 xen-01 ha-lizard:  TIMEELAPSED = 15
Dec 16 09:36:48 xen-01 ha-lizard:  MGT link failure duration = [ 15 ] Tolerance = [ 5 seconds ]
Dec 16 09:36:48 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:48 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:48 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:36:49 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:52 xen-01 iscsi-ha: 26069 Spawning new instance of iscsi-ha
Dec 16 09:36:52 xen-01 iscsi-ha: 26069 check_logger_processes Checking logger processes
Dec 16 09:36:52 xen-01 iscsi-ha: 26069 check_logger_processes No processes to clear
Dec 16 09:36:52 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:36:52 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:36:52 xen-01 iscsi-ha:  This iteration is count 6957
Dec 16 09:36:52 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:36:52 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 service_execute: System V mode detected
Dec 16 09:36:52 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 service_execute: Returning exit status [ 0 ]
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431696 nr:3152 dw:4369752 dr:1325208284 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:36:52 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:52 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 DRBD Resource: iscsi1 in Connected state
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 service_execute: systemctl mode being used
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 service_execute: Returning exit status [ 0 ]
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 CHECKING IP 127.0.0.1
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 CHECKING IP 172.16.14.199
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 CHECKING IP 172.16.14.47
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 CHECKING IP 10.10.10.1
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 CHECKING IP 10.10.10.3
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:36:52 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:36:52 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:36:52 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:36:52 xen-01 iscsi-ha: 26453 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:36:52 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:36:52 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:36:52 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:36:52 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:36:53 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:36:53 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:36:53 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:36:53 xen-01 ha-lizard:  TIMENOW = 1481881013
Dec 16 09:36:53 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:36:53 xen-01 ha-lizard:  TIMEELAPSED = 20
Dec 16 09:36:53 xen-01 ha-lizard:  MGT link failure duration = [ 20 ] Tolerance = [ 5 seconds ]
Dec 16 09:36:53 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:53 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:53 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:36:54 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:57 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 2 on PIDS: 25761
Dec 16 09:36:58 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:36:58 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:36:58 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:36:58 xen-01 ha-lizard:  TIMENOW = 1481881018
Dec 16 09:36:58 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:36:58 xen-01 ha-lizard:  TIMEELAPSED = 25
Dec 16 09:36:58 xen-01 ha-lizard:  MGT link failure duration = [ 25 ] Tolerance = [ 5 seconds ]
Dec 16 09:36:58 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:59 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:36:59 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:36:59 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:37:02 xen-01 iscsi-ha: 26439 Spawning new instance of iscsi-ha
Dec 16 09:37:02 xen-01 iscsi-ha: 26439 check_logger_processes Checking logger processes
Dec 16 09:37:02 xen-01 iscsi-ha: 26439 check_logger_processes No processes to clear
Dec 16 09:37:02 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:37:02 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:37:02 xen-01 iscsi-ha:  This iteration is count 6958
Dec 16 09:37:02 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:02 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 service_execute: System V mode detected
Dec 16 09:37:02 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 service_execute: Returning exit status [ 0 ]
Dec 16 09:37:02 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:02 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431700 nr:3152 dw:4369756 dr:1325208300 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 DRBD Resource: iscsi1 in Connected state
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 service_execute: systemctl mode being used
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 service_execute: Returning exit status [ 0 ]
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 CHECKING IP 127.0.0.1
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 CHECKING IP 172.16.14.199
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 CHECKING IP 172.16.14.47
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 CHECKING IP 10.10.10.1
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 CHECKING IP 10.10.10.3
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:37:02 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:37:02 xen-01 iscsi-ha: 26963 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:37:03 xen-01 iscsi-ha: 26963 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:37:03 xen-01 iscsi-ha: 26963 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:37:03 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:37:03 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:37:03 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:37:03 xen-01 iscsi-ha: 26963 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:37:03 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:37:03 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:37:03 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:37:04 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:04 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:37:04 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:37:04 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:37:04 xen-01 ha-lizard:  TIMENOW = 1481881024
Dec 16 09:37:04 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:37:04 xen-01 ha-lizard:  TIMEELAPSED = 31
Dec 16 09:37:04 xen-01 ha-lizard:  MGT link failure duration = [ 31 ] Tolerance = [ 5 seconds ]
Dec 16 09:37:04 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:04 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:04 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:37:07 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 3 on PIDS: 25761
Dec 16 09:37:09 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:09 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:37:09 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:37:09 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:37:09 xen-01 ha-lizard:  TIMENOW = 1481881029
Dec 16 09:37:09 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:37:09 xen-01 ha-lizard:  TIMEELAPSED = 36
Dec 16 09:37:09 xen-01 ha-lizard:  MGT link failure duration = [ 36 ] Tolerance = [ 5 seconds ]
Dec 16 09:37:09 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:09 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:09 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:37:12 xen-01 iscsi-ha: 26909 Spawning new instance of iscsi-ha
Dec 16 09:37:12 xen-01 iscsi-ha: 26909 check_logger_processes Checking logger processes
Dec 16 09:37:12 xen-01 iscsi-ha: 26909 check_logger_processes No processes to clear
Dec 16 09:37:12 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:37:12 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:37:12 xen-01 iscsi-ha:  This iteration is count 6959
Dec 16 09:37:12 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:12 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:37:12 xen-01 iscsi-ha: 28254 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:37:12 xen-01 iscsi-ha: 28254 service_execute: System V mode detected
Dec 16 09:37:12 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:12 xen-01 iscsi-ha: 28254 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:37:12 xen-01 iscsi-ha: 28254 service_execute: Returning exit status [ 0 ]
Dec 16 09:37:12 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:12 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:12 xen-01 iscsi-ha: 28254 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431700 nr:3152 dw:4369756 dr:1325208316 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:37:12 xen-01 iscsi-ha: 28254 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:37:12 xen-01 iscsi-ha: 28254 DRBD Resource: iscsi1 in Connected state
Dec 16 09:37:12 xen-01 iscsi-ha: 28254 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:37:12 xen-01 iscsi-ha: 28254 service_execute: systemctl mode being used
Dec 16 09:37:12 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:37:12 xen-01 iscsi-ha: 28254 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:37:12 xen-01 iscsi-ha: 28254 service_execute: Returning exit status [ 0 ]
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 18h ago ]
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 CHECKING IP 127.0.0.1
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 CHECKING IP 172.16.14.199
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 CHECKING IP 172.16.14.47
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 CHECKING IP 10.10.10.1
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 CHECKING IP 10.10.10.3
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:37:13 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:37:13 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:37:13 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:37:13 xen-01 iscsi-ha: 28254 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:37:13 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:37:13 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:37:13 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:37:14 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:14 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:37:14 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:37:14 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:37:14 xen-01 ha-lizard:  TIMENOW = 1481881034
Dec 16 09:37:14 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:37:14 xen-01 ha-lizard:  TIMEELAPSED = 41
Dec 16 09:37:14 xen-01 ha-lizard:  MGT link failure duration = [ 41 ] Tolerance = [ 5 seconds ]
Dec 16 09:37:14 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:14 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:14 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:37:17 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 4 on PIDS: 25761
Dec 16 09:37:19 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:19 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:37:19 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:37:19 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:37:19 xen-01 ha-lizard:  TIMENOW = 1481881039
Dec 16 09:37:19 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:37:19 xen-01 ha-lizard:  TIMEELAPSED = 46
Dec 16 09:37:19 xen-01 ha-lizard:  MGT link failure duration = [ 46 ] Tolerance = [ 5 seconds ]
Dec 16 09:37:19 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:19 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:19 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:37:22 xen-01 iscsi-ha: 28240 Spawning new instance of iscsi-ha
Dec 16 09:37:22 xen-01 iscsi-ha: 28240 check_logger_processes Checking logger processes
Dec 16 09:37:22 xen-01 iscsi-ha: 28240 check_logger_processes No processes to clear
Dec 16 09:37:22 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:37:22 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:37:22 xen-01 iscsi-ha:  This iteration is count 6960
Dec 16 09:37:22 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:22 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:37:22 xen-01 iscsi-ha: 28630 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:37:22 xen-01 iscsi-ha: 28630 service_execute: System V mode detected
Dec 16 09:37:22 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:22 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 service_execute: Returning exit status [ 0 ]
Dec 16 09:37:23 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:23 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431700 nr:3152 dw:4369756 dr:1325208332 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 DRBD Resource: iscsi1 in Connected state
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 service_execute: systemctl mode being used
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 service_execute: Returning exit status [ 0 ]
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 CHECKING IP 127.0.0.1
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 CHECKING IP 172.16.14.199
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 CHECKING IP 172.16.14.47
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 CHECKING IP 10.10.10.1
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 CHECKING IP 10.10.10.3
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:37:23 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:37:23 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:37:23 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:37:23 xen-01 iscsi-ha: 28630 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:37:23 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:37:23 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:37:23 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:37:24 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:24 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:37:25 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:37:25 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:37:25 xen-01 ha-lizard:  TIMENOW = 1481881045
Dec 16 09:37:25 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:37:25 xen-01 ha-lizard:  TIMEELAPSED = 52
Dec 16 09:37:25 xen-01 ha-lizard:  MGT link failure duration = [ 52 ] Tolerance = [ 5 seconds ]
Dec 16 09:37:25 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:25 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:25 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:37:27 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 5 on PIDS: 25761
Dec 16 09:37:29 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:30 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:37:30 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:37:30 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:37:30 xen-01 ha-lizard:  TIMENOW = 1481881050
Dec 16 09:37:30 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:37:30 xen-01 ha-lizard:  TIMEELAPSED = 57
Dec 16 09:37:30 xen-01 ha-lizard:  MGT link failure duration = [ 57 ] Tolerance = [ 5 seconds ]
Dec 16 09:37:30 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:30 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:30 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:37:32 xen-01 iscsi-ha: 28615 Spawning new instance of iscsi-ha
Dec 16 09:37:32 xen-01 iscsi-ha: 28615 check_logger_processes Checking logger processes
Dec 16 09:37:33 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:37:33 xen-01 iscsi-ha: 28615 check_logger_processes No processes to clear
Dec 16 09:37:33 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:37:33 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:37:33 xen-01 iscsi-ha:  This iteration is count 6961
Dec 16 09:37:33 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:33 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 service_execute: System V mode detected
Dec 16 09:37:33 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 service_execute: Returning exit status [ 0 ]
Dec 16 09:37:33 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:33 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431708 nr:3152 dw:4369764 dr:1325208348 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 DRBD Resource: iscsi1 in Connected state
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 service_execute: systemctl mode being used
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 service_execute: Returning exit status [ 0 ]
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 CHECKING IP 127.0.0.1
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 CHECKING IP 172.16.14.199
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 CHECKING IP 172.16.14.47
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 CHECKING IP 10.10.10.1
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 CHECKING IP 10.10.10.3
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:37:33 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:37:33 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:37:33 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:37:33 xen-01 iscsi-ha: 28966 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:37:33 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:37:33 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:37:33 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:37:34 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:35 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:37:35 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:37:35 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:37:35 xen-01 ha-lizard:  TIMENOW = 1481881055
Dec 16 09:37:35 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:37:35 xen-01 ha-lizard:  TIMEELAPSED = 62
Dec 16 09:37:35 xen-01 ha-lizard:  MGT link failure duration = [ 62 ] Tolerance = [ 5 seconds ]
Dec 16 09:37:35 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:35 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:35 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:37:37 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 6 on PIDS: 25761
Dec 16 09:37:39 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:40 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:37:40 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:37:40 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:37:40 xen-01 ha-lizard:  TIMENOW = 1481881060
Dec 16 09:37:40 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:37:40 xen-01 ha-lizard:  TIMEELAPSED = 67
Dec 16 09:37:40 xen-01 ha-lizard:  MGT link failure duration = [ 67 ] Tolerance = [ 5 seconds ]
Dec 16 09:37:40 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:40 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:40 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:37:43 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:37:43 xen-01 iscsi-ha: 28952 Spawning new instance of iscsi-ha
Dec 16 09:37:43 xen-01 iscsi-ha: 28952 check_logger_processes Checking logger processes
Dec 16 09:37:43 xen-01 iscsi-ha: 28952 check_logger_processes No processes to clear
Dec 16 09:37:43 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:37:43 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:37:43 xen-01 iscsi-ha:  This iteration is count 6962
Dec 16 09:37:43 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:43 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 service_execute: System V mode detected
Dec 16 09:37:43 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 service_execute: Returning exit status [ 0 ]
Dec 16 09:37:43 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:43 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431708 nr:3152 dw:4369764 dr:1325208364 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 DRBD Resource: iscsi1 in Connected state
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 service_execute: systemctl mode being used
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 service_execute: Returning exit status [ 0 ]
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 CHECKING IP 127.0.0.1
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 CHECKING IP 172.16.14.199
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 CHECKING IP 172.16.14.47
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 CHECKING IP 10.10.10.1
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 CHECKING IP 10.10.10.3
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:37:43 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:37:43 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:37:43 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:37:43 xen-01 iscsi-ha: 30409 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:37:43 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:37:43 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:37:43 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:37:44 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:45 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:37:45 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:37:45 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:37:45 xen-01 ha-lizard:  TIMENOW = 1481881065
Dec 16 09:37:45 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:37:45 xen-01 ha-lizard:  TIMEELAPSED = 72
Dec 16 09:37:45 xen-01 ha-lizard:  MGT link failure duration = [ 72 ] Tolerance = [ 5 seconds ]
Dec 16 09:37:45 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:46 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:46 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:37:48 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 7 on PIDS: 25761
Dec 16 09:37:49 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:51 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:37:51 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:37:51 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:37:51 xen-01 ha-lizard:  TIMENOW = 1481881071
Dec 16 09:37:51 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:37:51 xen-01 ha-lizard:  TIMEELAPSED = 78
Dec 16 09:37:51 xen-01 ha-lizard:  MGT link failure duration = [ 78 ] Tolerance = [ 5 seconds ]
Dec 16 09:37:51 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:51 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:51 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:37:53 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:37:53 xen-01 iscsi-ha: 30395 Spawning new instance of iscsi-ha
Dec 16 09:37:53 xen-01 iscsi-ha: 30395 check_logger_processes Checking logger processes
Dec 16 09:37:53 xen-01 iscsi-ha: 30395 check_logger_processes No processes to clear
Dec 16 09:37:53 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:37:53 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:37:53 xen-01 iscsi-ha:  This iteration is count 6963
Dec 16 09:37:53 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:37:53 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 service_execute: System V mode detected
Dec 16 09:37:53 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 service_execute: Returning exit status [ 0 ]
Dec 16 09:37:53 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:53 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431708 nr:3152 dw:4369764 dr:1325208380 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 DRBD Resource: iscsi1 in Connected state
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 service_execute: systemctl mode being used
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 service_execute: Returning exit status [ 0 ]
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 CHECKING IP 127.0.0.1
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 CHECKING IP 172.16.14.199
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 CHECKING IP 172.16.14.47
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 CHECKING IP 10.10.10.1
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 CHECKING IP 10.10.10.3
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:37:53 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:37:53 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:37:53 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:37:53 xen-01 iscsi-ha: 30781 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:37:53 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:37:53 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:37:53 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:37:54 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:37:56 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:37:56 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:37:56 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:37:56 xen-01 ha-lizard:  TIMENOW = 1481881076
Dec 16 09:37:56 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:37:56 xen-01 ha-lizard:  TIMEELAPSED = 83
Dec 16 09:37:56 xen-01 ha-lizard:  MGT link failure duration = [ 83 ] Tolerance = [ 5 seconds ]
Dec 16 09:37:56 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:56 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:37:56 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:37:58 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 8 on PIDS: 25761
Dec 16 09:37:59 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:01 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:38:01 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:38:01 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:38:01 xen-01 ha-lizard:  TIMENOW = 1481881081
Dec 16 09:38:01 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:38:01 xen-01 ha-lizard:  TIMEELAPSED = 88
Dec 16 09:38:01 xen-01 ha-lizard:  MGT link failure duration = [ 88 ] Tolerance = [ 5 seconds ]
Dec 16 09:38:01 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:01 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:01 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:38:03 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:38:03 xen-01 iscsi-ha: 30767 Spawning new instance of iscsi-ha
Dec 16 09:38:03 xen-01 iscsi-ha: 30767 check_logger_processes Checking logger processes
Dec 16 09:38:03 xen-01 iscsi-ha: 30767 check_logger_processes No processes to clear
Dec 16 09:38:03 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:38:03 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:38:03 xen-01 iscsi-ha:  This iteration is count 6964
Dec 16 09:38:03 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:03 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 service_execute: System V mode detected
Dec 16 09:38:03 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 service_execute: Returning exit status [ 0 ]
Dec 16 09:38:03 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:03 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431712 nr:3152 dw:4369768 dr:1325208396 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 DRBD Resource: iscsi1 in Connected state
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 service_execute: systemctl mode being used
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 service_execute: Returning exit status [ 0 ]
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 CHECKING IP 127.0.0.1
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 CHECKING IP 172.16.14.199
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 CHECKING IP 172.16.14.47
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 CHECKING IP 10.10.10.1
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 CHECKING IP 10.10.10.3
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:38:03 xen-01 iscsi-ha: 31109 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:38:03 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:38:03 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:38:03 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:38:04 xen-01 iscsi-ha: 31109 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:38:04 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:38:04 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:38:04 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:38:04 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:06 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:38:06 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:38:06 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:38:06 xen-01 ha-lizard:  TIMENOW = 1481881086
Dec 16 09:38:06 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:38:06 xen-01 ha-lizard:  TIMEELAPSED = 93
Dec 16 09:38:06 xen-01 ha-lizard:  MGT link failure duration = [ 93 ] Tolerance = [ 5 seconds ]
Dec 16 09:38:06 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:06 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:06 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:38:08 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 9 on PIDS: 25761
Dec 16 09:38:09 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:11 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:38:12 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:38:12 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:38:12 xen-01 ha-lizard:  TIMENOW = 1481881092
Dec 16 09:38:12 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:38:12 xen-01 ha-lizard:  TIMEELAPSED = 99
Dec 16 09:38:12 xen-01 ha-lizard:  MGT link failure duration = [ 99 ] Tolerance = [ 5 seconds ]
Dec 16 09:38:12 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:12 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:12 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:38:13 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:38:13 xen-01 iscsi-ha: 31093 Spawning new instance of iscsi-ha
Dec 16 09:38:13 xen-01 iscsi-ha: 31093 check_logger_processes Checking logger processes
Dec 16 09:38:13 xen-01 iscsi-ha: 31093 check_logger_processes No processes to clear
Dec 16 09:38:13 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:38:13 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:38:13 xen-01 iscsi-ha:  This iteration is count 6965
Dec 16 09:38:13 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:13 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:38:13 xen-01 iscsi-ha: 32152 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:38:13 xen-01 iscsi-ha: 32152 service_execute: System V mode detected
Dec 16 09:38:13 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:13 xen-01 iscsi-ha: 32152 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:38:13 xen-01 iscsi-ha: 32152 service_execute: Returning exit status [ 0 ]
Dec 16 09:38:13 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:13 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:13 xen-01 iscsi-ha: 32152 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431712 nr:3152 dw:4369768 dr:1325208412 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:38:13 xen-01 iscsi-ha: 32152 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:38:13 xen-01 iscsi-ha: 32152 DRBD Resource: iscsi1 in Connected state
Dec 16 09:38:13 xen-01 iscsi-ha: 32152 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:38:13 xen-01 iscsi-ha: 32152 service_execute: systemctl mode being used
Dec 16 09:38:13 xen-01 iscsi-ha: 32152 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:38:13 xen-01 iscsi-ha: 32152 service_execute: Returning exit status [ 0 ]
Dec 16 09:38:13 xen-01 iscsi-ha: 32152 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:38:14 xen-01 iscsi-ha: 32152 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:38:14 xen-01 iscsi-ha: 32152 CHECKING IP 127.0.0.1
Dec 16 09:38:14 xen-01 iscsi-ha: 32152 CHECKING IP 172.16.14.199
Dec 16 09:38:14 xen-01 iscsi-ha: 32152 CHECKING IP 172.16.14.47
Dec 16 09:38:14 xen-01 iscsi-ha: 32152 CHECKING IP 10.10.10.1
Dec 16 09:38:14 xen-01 iscsi-ha: 32152 CHECKING IP 10.10.10.3
Dec 16 09:38:14 xen-01 iscsi-ha: 32152 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:38:14 xen-01 iscsi-ha: 32152 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:38:14 xen-01 iscsi-ha: 32152 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:38:14 xen-01 iscsi-ha: 32152 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:38:14 xen-01 iscsi-ha: 32152 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:38:14 xen-01 iscsi-ha: 32152 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:38:14 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:38:14 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:38:14 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:38:14 xen-01 iscsi-ha: 32152 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:38:14 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:38:14 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:38:14 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:38:14 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:17 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:38:17 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:38:17 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:38:17 xen-01 ha-lizard:  TIMENOW = 1481881097
Dec 16 09:38:17 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:38:17 xen-01 ha-lizard:  TIMEELAPSED = 104
Dec 16 09:38:17 xen-01 ha-lizard:  MGT link failure duration = [ 104 ] Tolerance = [ 5 seconds ]
Dec 16 09:38:17 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:17 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:17 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:38:18 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 10 on PIDS: 25761
Dec 16 09:38:19 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:22 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:38:22 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:38:22 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:38:22 xen-01 ha-lizard:  TIMENOW = 1481881102
Dec 16 09:38:22 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:38:22 xen-01 ha-lizard:  TIMEELAPSED = 109
Dec 16 09:38:22 xen-01 ha-lizard:  MGT link failure duration = [ 109 ] Tolerance = [ 5 seconds ]
Dec 16 09:38:22 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:22 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:22 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:38:23 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:38:23 xen-01 iscsi-ha: 32127 Spawning new instance of iscsi-ha
Dec 16 09:38:23 xen-01 iscsi-ha: 32127 check_logger_processes Checking logger processes
Dec 16 09:38:23 xen-01 iscsi-ha: 32127 check_logger_processes No processes to clear
Dec 16 09:38:23 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:38:23 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:38:23 xen-01 iscsi-ha:  This iteration is count 6966
Dec 16 09:38:23 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:23 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:38:23 xen-01 iscsi-ha: 453 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:38:23 xen-01 iscsi-ha: 453 service_execute: System V mode detected
Dec 16 09:38:23 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:24 xen-01 iscsi-ha: 453 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:38:24 xen-01 iscsi-ha: 453 service_execute: Returning exit status [ 0 ]
Dec 16 09:38:24 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:24 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:24 xen-01 iscsi-ha: 453 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431712 nr:3152 dw:4369768 dr:1325208428 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:38:24 xen-01 iscsi-ha: 453 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:38:24 xen-01 iscsi-ha: 453 DRBD Resource: iscsi1 in Connected state
Dec 16 09:38:24 xen-01 iscsi-ha: 453 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:38:24 xen-01 iscsi-ha: 453 service_execute: systemctl mode being used
Dec 16 09:38:24 xen-01 iscsi-ha: 453 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:38:24 xen-01 iscsi-ha: 453 service_execute: Returning exit status [ 0 ]
Dec 16 09:38:24 xen-01 iscsi-ha: 453 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:38:24 xen-01 iscsi-ha: 453 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:38:24 xen-01 iscsi-ha: 453 CHECKING IP 127.0.0.1
Dec 16 09:38:24 xen-01 iscsi-ha: 453 CHECKING IP 172.16.14.199
Dec 16 09:38:24 xen-01 iscsi-ha: 453 CHECKING IP 172.16.14.47
Dec 16 09:38:24 xen-01 iscsi-ha: 453 CHECKING IP 10.10.10.1
Dec 16 09:38:24 xen-01 iscsi-ha: 453 CHECKING IP 10.10.10.3
Dec 16 09:38:24 xen-01 iscsi-ha: 453 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:38:24 xen-01 iscsi-ha: 453 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:38:24 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:24 xen-01 iscsi-ha: 453 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:38:24 xen-01 iscsi-ha: 453 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:38:24 xen-01 iscsi-ha: 453 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:38:24 xen-01 iscsi-ha: 453 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:38:24 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:38:24 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:38:24 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:38:24 xen-01 iscsi-ha: 453 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:38:24 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:38:24 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:38:24 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:38:27 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:38:27 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:38:27 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:38:27 xen-01 ha-lizard:  TIMENOW = 1481881107
Dec 16 09:38:27 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:38:27 xen-01 ha-lizard:  TIMEELAPSED = 114
Dec 16 09:38:27 xen-01 ha-lizard:  MGT link failure duration = [ 114 ] Tolerance = [ 5 seconds ]
Dec 16 09:38:27 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:27 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:27 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:38:28 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 11 on PIDS: 25761
Dec 16 09:38:29 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:32 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:38:32 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:38:32 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:38:32 xen-01 ha-lizard:  TIMENOW = 1481881112
Dec 16 09:38:32 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:38:32 xen-01 ha-lizard:  TIMEELAPSED = 119
Dec 16 09:38:32 xen-01 ha-lizard:  MGT link failure duration = [ 119 ] Tolerance = [ 5 seconds ]
Dec 16 09:38:32 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:32 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:33 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:38:33 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:38:33 xen-01 iscsi-ha: 439 Spawning new instance of iscsi-ha
Dec 16 09:38:33 xen-01 iscsi-ha: 439 check_logger_processes Checking logger processes
Dec 16 09:38:34 xen-01 iscsi-ha: 439 check_logger_processes No processes to clear
Dec 16 09:38:34 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:38:34 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:38:34 xen-01 iscsi-ha:  This iteration is count 6967
Dec 16 09:38:34 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:34 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:38:34 xen-01 iscsi-ha: 777 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:38:34 xen-01 iscsi-ha: 777 service_execute: System V mode detected
Dec 16 09:38:34 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:34 xen-01 iscsi-ha: 777 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:38:34 xen-01 iscsi-ha: 777 service_execute: Returning exit status [ 0 ]
Dec 16 09:38:34 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:34 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:34 xen-01 iscsi-ha: 777 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431720 nr:3152 dw:4369776 dr:1325208444 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:38:34 xen-01 iscsi-ha: 777 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:38:34 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:34 xen-01 iscsi-ha: 777 DRBD Resource: iscsi1 in Connected state
Dec 16 09:38:34 xen-01 iscsi-ha: 777 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:38:34 xen-01 iscsi-ha: 777 service_execute: systemctl mode being used
Dec 16 09:38:34 xen-01 iscsi-ha: 777 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:38:34 xen-01 iscsi-ha: 777 service_execute: Returning exit status [ 0 ]
Dec 16 09:38:34 xen-01 iscsi-ha: 777 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:38:34 xen-01 iscsi-ha: 777 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:38:34 xen-01 iscsi-ha: 777 CHECKING IP 127.0.0.1
Dec 16 09:38:34 xen-01 iscsi-ha: 777 CHECKING IP 172.16.14.199
Dec 16 09:38:34 xen-01 iscsi-ha: 777 CHECKING IP 172.16.14.47
Dec 16 09:38:34 xen-01 iscsi-ha: 777 CHECKING IP 10.10.10.1
Dec 16 09:38:34 xen-01 iscsi-ha: 777 CHECKING IP 10.10.10.3
Dec 16 09:38:34 xen-01 iscsi-ha: 777 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:38:34 xen-01 iscsi-ha: 777 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:38:34 xen-01 iscsi-ha: 777 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:38:34 xen-01 iscsi-ha: 777 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:38:34 xen-01 iscsi-ha: 777 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:38:34 xen-01 iscsi-ha: 777 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:38:34 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:38:34 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:38:34 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:38:34 xen-01 iscsi-ha: 777 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:38:34 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:38:34 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:38:34 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:38:38 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:38:38 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:38:38 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:38:38 xen-01 ha-lizard:  TIMENOW = 1481881118
Dec 16 09:38:38 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:38:38 xen-01 ha-lizard:  TIMEELAPSED = 125
Dec 16 09:38:38 xen-01 ha-lizard:  MGT link failure duration = [ 125 ] Tolerance = [ 5 seconds ]
Dec 16 09:38:38 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:38 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:38 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:38:38 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 12 on PIDS: 25761
Dec 16 09:38:39 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:43 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:38:43 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:38:43 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:38:43 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:38:43 xen-01 ha-lizard:  TIMENOW = 1481881123
Dec 16 09:38:43 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:38:43 xen-01 ha-lizard:  TIMEELAPSED = 130
Dec 16 09:38:43 xen-01 ha-lizard:  MGT link failure duration = [ 130 ] Tolerance = [ 5 seconds ]
Dec 16 09:38:43 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:43 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:43 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:38:44 xen-01 iscsi-ha: 761 Spawning new instance of iscsi-ha
Dec 16 09:38:44 xen-01 iscsi-ha: 761 check_logger_processes Checking logger processes
Dec 16 09:38:44 xen-01 iscsi-ha: 761 check_logger_processes No processes to clear
Dec 16 09:38:44 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:38:44 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:38:44 xen-01 iscsi-ha:  This iteration is count 6968
Dec 16 09:38:44 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:44 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 service_execute: System V mode detected
Dec 16 09:38:44 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:44 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 service_execute: Returning exit status [ 0 ]
Dec 16 09:38:44 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:44 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431720 nr:3152 dw:4369776 dr:1325208460 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 DRBD Resource: iscsi1 in Connected state
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 service_execute: systemctl mode being used
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 service_execute: Returning exit status [ 0 ]
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 CHECKING IP 127.0.0.1
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 CHECKING IP 172.16.14.199
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 CHECKING IP 172.16.14.47
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 CHECKING IP 10.10.10.1
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 CHECKING IP 10.10.10.3
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:38:44 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:38:44 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:38:44 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:38:44 xen-01 iscsi-ha: 1347 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:38:44 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:38:44 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:38:44 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:38:48 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:38:48 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 13 on PIDS: 25761
Dec 16 09:38:48 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:38:48 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:38:48 xen-01 ha-lizard:  TIMENOW = 1481881128
Dec 16 09:38:48 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:38:48 xen-01 ha-lizard:  TIMEELAPSED = 135
Dec 16 09:38:48 xen-01 ha-lizard:  MGT link failure duration = [ 135 ] Tolerance = [ 5 seconds ]
Dec 16 09:38:48 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:48 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:48 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:38:49 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:53 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:38:53 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:38:53 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:38:53 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:38:53 xen-01 ha-lizard:  TIMENOW = 1481881133
Dec 16 09:38:53 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:38:53 xen-01 ha-lizard:  TIMEELAPSED = 140
Dec 16 09:38:53 xen-01 ha-lizard:  MGT link failure duration = [ 140 ] Tolerance = [ 5 seconds ]
Dec 16 09:38:53 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:53 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:53 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:38:54 xen-01 iscsi-ha: 1331 Spawning new instance of iscsi-ha
Dec 16 09:38:54 xen-01 iscsi-ha: 1331 check_logger_processes Checking logger processes
Dec 16 09:38:54 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:38:54 xen-01 iscsi-ha: 1331 check_logger_processes No processes to clear
Dec 16 09:38:54 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:38:54 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:38:54 xen-01 iscsi-ha:  This iteration is count 6969
Dec 16 09:38:54 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:38:54 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 service_execute: System V mode detected
Dec 16 09:38:54 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 service_execute: Returning exit status [ 0 ]
Dec 16 09:38:54 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:54 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431720 nr:3152 dw:4369776 dr:1325208476 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 DRBD Resource: iscsi1 in Connected state
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 service_execute: systemctl mode being used
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 service_execute: Returning exit status [ 0 ]
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 CHECKING IP 127.0.0.1
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 CHECKING IP 172.16.14.199
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 CHECKING IP 172.16.14.47
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 CHECKING IP 10.10.10.1
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 CHECKING IP 10.10.10.3
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:38:54 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:38:54 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:38:54 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:38:54 xen-01 iscsi-ha: 2994 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:38:54 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:38:54 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:38:54 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:38:58 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 14 on PIDS: 25761
Dec 16 09:38:58 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:38:59 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:38:59 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:38:59 xen-01 ha-lizard:  TIMENOW = 1481881139
Dec 16 09:38:59 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:38:59 xen-01 ha-lizard:  TIMEELAPSED = 146
Dec 16 09:38:59 xen-01 ha-lizard:  MGT link failure duration = [ 146 ] Tolerance = [ 5 seconds ]
Dec 16 09:38:59 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:59 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:38:59 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:38:59 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:03 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:39:04 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:39:04 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:39:04 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:39:04 xen-01 ha-lizard:  TIMENOW = 1481881144
Dec 16 09:39:04 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:39:04 xen-01 ha-lizard:  TIMEELAPSED = 151
Dec 16 09:39:04 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:04 xen-01 ha-lizard:  MGT link failure duration = [ 151 ] Tolerance = [ 5 seconds ]
Dec 16 09:39:04 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:39:04 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:39:04 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:39:04 xen-01 iscsi-ha: 2979 Spawning new instance of iscsi-ha
Dec 16 09:39:04 xen-01 iscsi-ha: 2979 check_logger_processes Checking logger processes
Dec 16 09:39:04 xen-01 iscsi-ha: 2979 check_logger_processes No processes to clear
Dec 16 09:39:04 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:39:04 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:39:04 xen-01 iscsi-ha:  This iteration is count 6970
Dec 16 09:39:04 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:04 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 service_execute: System V mode detected
Dec 16 09:39:04 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 service_execute: Returning exit status [ 0 ]
Dec 16 09:39:04 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:04 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431724 nr:3152 dw:4369780 dr:1325208492 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 DRBD Resource: iscsi1 in Connected state
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 service_execute: systemctl mode being used
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 service_execute: Returning exit status [ 0 ]
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 CHECKING IP 127.0.0.1
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 CHECKING IP 172.16.14.199
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 CHECKING IP 172.16.14.47
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 CHECKING IP 10.10.10.1
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 CHECKING IP 10.10.10.3
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:39:04 xen-01 iscsi-ha: 3322 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:39:04 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:39:04 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:39:04 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:39:05 xen-01 iscsi-ha: 3322 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:39:05 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:39:05 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:39:05 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:39:08 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 15 on PIDS: 25761
Dec 16 09:39:09 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:09 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:39:09 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:39:09 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:39:09 xen-01 ha-lizard:  TIMENOW = 1481881149
Dec 16 09:39:09 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:39:09 xen-01 ha-lizard:  TIMEELAPSED = 156
Dec 16 09:39:09 xen-01 ha-lizard:  MGT link failure duration = [ 156 ] Tolerance = [ 5 seconds ]
Dec 16 09:39:09 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:39:09 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:39:09 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:39:13 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:39:14 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:14 xen-01 iscsi-ha: 3306 Spawning new instance of iscsi-ha
Dec 16 09:39:14 xen-01 iscsi-ha: 3306 check_logger_processes Checking logger processes
Dec 16 09:39:14 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:39:14 xen-01 iscsi-ha: 3306 check_logger_processes No processes to clear
Dec 16 09:39:14 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:39:14 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:39:14 xen-01 iscsi-ha:  This iteration is count 6971
Dec 16 09:39:14 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:14 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 service_execute: System V mode detected
Dec 16 09:39:14 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:39:14 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:39:14 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:14 xen-01 ha-lizard:  TIMENOW = 1481881154
Dec 16 09:39:14 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:39:14 xen-01 ha-lizard:  TIMEELAPSED = 161
Dec 16 09:39:14 xen-01 ha-lizard:  MGT link failure duration = [ 161 ] Tolerance = [ 5 seconds ]
Dec 16 09:39:14 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:39:14 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 service_execute: Returning exit status [ 0 ]
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431724 nr:3152 dw:4369780 dr:1325208492 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:39:14 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 DRBD Resource: iscsi1 in Connected state
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 service_execute: systemctl mode being used
Dec 16 09:39:14 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:14 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 service_execute: Returning exit status [ 0 ]
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 CHECKING IP 127.0.0.1
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 CHECKING IP 172.16.14.199
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 CHECKING IP 172.16.14.47
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 CHECKING IP 10.10.10.1
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 CHECKING IP 10.10.10.3
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:39:14 xen-01 iscsi-ha: 3818 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:39:15 xen-01 iscsi-ha: 3818 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:39:15 xen-01 iscsi-ha: 3818 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:39:15 xen-01 iscsi-ha: 3818 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:39:15 xen-01 iscsi-ha: 3818 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:39:15 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:39:15 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:39:15 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:39:15 xen-01 iscsi-ha: 3818 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:39:15 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:39:15 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:39:15 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:39:18 xen-01 ha-lizard: 25761 ha-lizard already running: Attempt 16 on PIDS: 25761
Dec 16 09:39:19 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:19 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:39:20 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ false ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:39:20 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state DOWN
Dec 16 09:39:20 xen-01 ha-lizard:  TIMENOW = 1481881160
Dec 16 09:39:20 xen-01 ha-lizard:  TIMEFAILED = 1481880993
Dec 16 09:39:20 xen-01 ha-lizard:  TIMEELAPSED = 167
Dec 16 09:39:20 xen-01 ha-lizard:  MGT link failure duration = [ 167 ] Tolerance = [ 5 seconds ]
Dec 16 09:39:20 xen-01 ha-lizard:  Management link outage tolerance [ 5 seconds ] reached - shutting down ALL VMs on Master [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:39:20 xen-01 ha-lizard:  stop_vms_on_host: Stopping all VMs on host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ]
Dec 16 09:39:20 xen-01 ha-lizard:  MGT link is down - waiting for link to be restored
Dec 16 09:39:23 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:39:24 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:24 xen-01 iscsi-ha: 3790 Spawning new instance of iscsi-ha
Dec 16 09:39:24 xen-01 iscsi-ha: 3790 check_logger_processes Checking logger processes
Dec 16 09:39:24 xen-01 cmanicd: NIC agent link is up : eth4
Dec 16 09:39:24 xen-01 iscsi-ha: 3790 check_logger_processes No processes to clear
Dec 16 09:39:24 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:39:24 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:39:24 xen-01 iscsi-ha:  This iteration is count 6972
Dec 16 09:39:24 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:24 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:39:24 xen-01 iscsi-ha: 5086 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:39:24 xen-01 iscsi-ha: 5086 service_execute: System V mode detected
Dec 16 09:39:24 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 service_execute: Returning exit status [ 0 ]
Dec 16 09:39:25 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:25 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431724 nr:3152 dw:4369780 dr:1325208524 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 DRBD Resource: iscsi1 in Connected state
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 service_execute: systemctl mode being used
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 service_execute: Returning exit status [ 0 ]
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 CHECKING IP 127.0.0.1
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 CHECKING IP 172.16.14.199
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 CHECKING IP 172.16.14.47
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 CHECKING IP 10.10.10.1
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 CHECKING IP 10.10.10.3
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:39:25 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:39:25 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:39:25 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:39:25 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:39:25 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:39:25 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:39:25 xen-01 ha-lizard:  MGT link has been restored
Dec 16 09:39:25 xen-01 iscsi-ha: 5086 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:39:25 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:39:25 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:39:25 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:39:28 xen-01 ha-lizard: 25761 Spawning new instance of ha-lizard
Dec 16 09:39:28 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:39:28 xen-01 ha-lizard:  This iteration is count 4631
Dec 16 09:39:28 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:28 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:39:29 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:39:29 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:39:29 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:39:29 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:39:29 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:39:29 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:39:29 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:39:29 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:39:29 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:39:29 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:29 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:39:29 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:39:29 xen-01 ha-lizard:  Master management link OK - checking prior link state
Dec 16 09:39:29 xen-01 ha-lizard:  Management link transitioned from DOWN -> UP
Dec 16 09:39:29 xen-01 ha-lizard:  Master sleep for XAPI_COUNT [ 2 ] [X] XAPI_DELAY [ 10 ] + 10
Dec 16 09:39:29 xen-01 ha-lizard:  Delaying master execution [ 30 ] seconds
Dec 16 09:39:33 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:39:34 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:34 xen-01 iscsi-ha: 5072 Spawning new instance of iscsi-ha
Dec 16 09:39:34 xen-01 iscsi-ha: 5072 check_logger_processes Checking logger processes
Dec 16 09:39:34 xen-01 iscsi-ha: 5072 check_logger_processes No processes to clear
Dec 16 09:39:35 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:39:35 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:39:35 xen-01 iscsi-ha:  This iteration is count 6973
Dec 16 09:39:35 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:35 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 service_execute: System V mode detected
Dec 16 09:39:35 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 service_execute: Returning exit status [ 0 ]
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431732 nr:3152 dw:4369788 dr:1325208524 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 DRBD Resource: iscsi1 in Connected state
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 service_execute: systemctl mode being used
Dec 16 09:39:35 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:35 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 service_execute: Returning exit status [ 0 ]
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 CHECKING IP 127.0.0.1
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 CHECKING IP 172.16.14.199
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 CHECKING IP 172.16.14.47
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 CHECKING IP 10.10.10.1
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 CHECKING IP 10.10.10.3
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:39:35 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:39:35 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:39:35 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:39:35 xen-01 iscsi-ha: 5491 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:39:35 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:39:35 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:39:35 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:39:39 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:43 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:39:43 xen-01 ha-lizard: 5255 ha-lizard already running: Attempt 1 on PIDS: 5255
Dec 16 09:39:44 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:45 xen-01 iscsi-ha: 5477 Spawning new instance of iscsi-ha
Dec 16 09:39:45 xen-01 iscsi-ha: 5477 check_logger_processes Checking logger processes
Dec 16 09:39:45 xen-01 iscsi-ha: 5477 check_logger_processes No processes to clear
Dec 16 09:39:45 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:39:45 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:39:45 xen-01 iscsi-ha:  This iteration is count 6974
Dec 16 09:39:45 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:45 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 service_execute: System V mode detected
Dec 16 09:39:45 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 service_execute: Returning exit status [ 0 ]
Dec 16 09:39:45 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:45 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431732 nr:3152 dw:4369788 dr:1325208556 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 DRBD Resource: iscsi1 in Connected state
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 service_execute: systemctl mode being used
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 service_execute: Returning exit status [ 0 ]
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 CHECKING IP 127.0.0.1
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 CHECKING IP 172.16.14.199
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 CHECKING IP 172.16.14.47
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 CHECKING IP 10.10.10.1
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 CHECKING IP 10.10.10.3
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:39:45 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:39:45 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:39:45 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:39:45 xen-01 iscsi-ha: 5921 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:39:45 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:39:45 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:39:45 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:39:49 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:53 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:39:54 xen-01 ha-lizard: 5255 ha-lizard already running: Attempt 2 on PIDS: 5255
Dec 16 09:39:54 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:55 xen-01 iscsi-ha: 5907 Spawning new instance of iscsi-ha
Dec 16 09:39:55 xen-01 iscsi-ha: 5907 check_logger_processes Checking logger processes
Dec 16 09:39:55 xen-01 iscsi-ha: 5907 check_logger_processes No processes to clear
Dec 16 09:39:55 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:39:55 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:39:55 xen-01 iscsi-ha:  This iteration is count 6975
Dec 16 09:39:55 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:39:55 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 service_execute: System V mode detected
Dec 16 09:39:55 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 service_execute: Returning exit status [ 0 ]
Dec 16 09:39:55 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:55 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431732 nr:3152 dw:4369788 dr:1325208572 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 DRBD Resource: iscsi1 in Connected state
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 service_execute: systemctl mode being used
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 service_execute: Returning exit status [ 0 ]
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 CHECKING IP 127.0.0.1
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 CHECKING IP 172.16.14.199
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 CHECKING IP 172.16.14.47
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 CHECKING IP 10.10.10.1
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 CHECKING IP 10.10.10.3
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:39:55 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:39:55 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:39:55 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:39:55 xen-01 iscsi-ha: 6322 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:39:55 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:39:55 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:39:55 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:39:59 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:39:59 xen-01 ha-lizard:  service_execute: Execute [ restart ] on [ xapi ]
Dec 16 09:39:59 xen-01 ha-lizard:  service_execute: System V mode detected
Dec 16 09:40:03 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:40:03 xen-01 logger: xapi stopped successfully
Dec 16 09:40:04 xen-01 ha-lizard:  service_execute: Restarting xapi (via systemctl):  [  OK  ]
Dec 16 09:40:04 xen-01 ha-lizard:  service_execute: Returning exit status [ 0 ]
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Restarting xapi (via systemctl):  [  OK  ]
Dec 16 09:40:04 xen-01 ha-lizard: 5255 Spawning new instance of ha-lizard
Dec 16 09:40:04 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:04 xen-01 ha-lizard:  This iteration is count 4632
Dec 16 09:40:04 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:04 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:40:04 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error: Connection refused (calling connect )
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Error: Connection refused (calling connect )
Dec 16 09:40:04 xen-01 ha-lizard:  check_ha_enabled: Error retrieving pool uuid while checking if ha-lizard is enabled
Dec 16 09:40:04 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:04 xen-01 ha-lizard:  check_email_enabled: Email enabled for check_ha_enabled
Dec 16 09:40:04 xen-01 ha-lizard:  email: Sending ALERT email to root@localhost: check_ha_enabled:  Error retrieving pool uuid while checking if ha-lizard is enabled- major error - ha-lizard in unknown state
Dec 16 09:40:04 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Traceback (most recent call last):
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email from: root@localhost
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/etc/ha-lizard/scripts/email_alert.py", line 105, in <module>
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email to: root@localhost
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xen-01
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     message = smtplib.SMTP(smtp_server, smtp_port, hostname)
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Timestamp: Fri Dec 16 09:40:04 GMT 2016
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Process: ha-lizard Version: 2.1.3
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Content: check_ha_enabled: Error retrieving pool uuid while checking if ha-lizard is enabled- major error - ha-lizard in unknown state
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: 
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     (code, msg) = self.connect(host, port)
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 315, in connect
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool UUID       : e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Name-Label : XenPool
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master UUID: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     self.sock = self._get_socket(host, port, self.timeout)
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-01
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : true
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     return socket.create_connection((host, port), timeout)
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-02
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : false
Dec 16 09:40:04 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Hostname: xen-01
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     raise err
Dec 16 09:40:04 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: socket.error: [Errno 111] Connection refused
Dec 16 09:40:04 xen-01 ha-lizard:  email: Message copied to /dev/shm/ha-lizard-mail/1481881204.msg - Suppressing duplicates for 60 Minutes
Dec 16 09:40:05 xen-01 iscsi-ha: 6298 Spawning new instance of iscsi-ha
Dec 16 09:40:05 xen-01 iscsi-ha: 6298 check_logger_processes Checking logger processes
Dec 16 09:40:05 xen-01 iscsi-ha: 6298 check_logger_processes No processes to clear
Dec 16 09:40:05 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:40:05 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:05 xen-01 iscsi-ha:  This iteration is count 6976
Dec 16 09:40:05 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:05 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 service_execute: System V mode detected
Dec 16 09:40:05 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: Error: Connection refused (calling connect )
Dec 16 09:40:05 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List:
Dec 16 09:40:05 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: Error: Connection refused (calling connect )
Dec 16 09:40:05 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: Error: Connection refused (calling connect )
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 service_execute: Returning exit status [ 0 ]
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431736 nr:3152 dw:4369792 dr:1325208588 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 DRBD Resource: iscsi1 in Connected state
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 service_execute: systemctl mode being used
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 service_execute: Returning exit status [ 0 ]
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 CHECKING IP 127.0.0.1
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 CHECKING IP 172.16.14.199
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 CHECKING IP 172.16.14.47
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 CHECKING IP 10.10.10.1
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 CHECKING IP 10.10.10.3
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:40:05 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: Error: Connection refused (calling connect )
Dec 16 09:40:05 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: message repeated 3 times: [ Error: Connection refused (calling connect )]
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 send_network_bridge_arp: Checking if interface is a bond
Dec 16 09:40:05 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: Error: Connection refused (calling connect )
Dec 16 09:40:05 xen-01 iscsi-ha: 7710 send_network_bridge_arp: ERROR detecting link interface(s) for [ xenbr6 ]
Dec 16 09:40:09 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:13 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:40:14 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:15 xen-01 iscsi-ha: 7696 Spawning new instance of iscsi-ha
Dec 16 09:40:15 xen-01 iscsi-ha: 7696 check_logger_processes Checking logger processes
Dec 16 09:40:15 xen-01 iscsi-ha: 7696 check_logger_processes No processes to clear
Dec 16 09:40:15 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:40:15 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:15 xen-01 iscsi-ha:  This iteration is count 6977
Dec 16 09:40:15 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:15 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 service_execute: System V mode detected
Dec 16 09:40:15 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 service_execute: Returning exit status [ 0 ]
Dec 16 09:40:15 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:15 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431736 nr:3152 dw:4369792 dr:1325208604 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 DRBD Resource: iscsi1 in Connected state
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 service_execute: systemctl mode being used
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 service_execute: Returning exit status [ 0 ]
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 CHECKING IP 127.0.0.1
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 CHECKING IP 172.16.14.199
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 CHECKING IP 172.16.14.47
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 CHECKING IP 10.10.10.1
Dec 16 09:40:15 xen-01 iscsi-ha: 8434 CHECKING IP 10.10.10.3
Dec 16 09:40:16 xen-01 iscsi-ha: 8434 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:40:16 xen-01 iscsi-ha: 8434 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:40:16 xen-01 iscsi-ha: 8434 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:40:16 xen-01 iscsi-ha: 8434 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:40:16 xen-01 iscsi-ha: 8434 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:40:16 xen-01 iscsi-ha: 8434 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:40:16 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:40:16 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:40:16 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:40:16 xen-01 iscsi-ha: 8434 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:40:16 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:40:16 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:40:16 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:40:19 xen-01 ha-lizard: 7543 Spawning new instance of ha-lizard
Dec 16 09:40:19 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:19 xen-01 ha-lizard:  This iteration is count 4633
Dec 16 09:40:19 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:19 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:40:19 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:40:19 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:19 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:19 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:40:19 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:40:19 xen-01 ha-lizard:  check_ha_enabled: Host is in maintenance mode
Dec 16 09:40:19 xen-01 ha-lizard:  check_ha_enabled: Master cannot be in maintenance mode - enabling host
Dec 16 09:40:19 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait.
Dec 16 09:40:19 xen-01 ha-lizard:  check_ha_enabled: Host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ] has been enabled
Dec 16 09:40:19 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:40:19 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:40:19 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:40:19 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:40:19 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:40:19 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:40:19 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:40:19 xen-01 ha-lizard:  Master management link OK - checking prior link state
Dec 16 09:40:19 xen-01 ha-lizard:  This host detected as pool  Master
Dec 16 09:40:19 xen-01 ha-lizard:  Found 2 hosts in pool
Dec 16 09:40:19 xen-01 ha-lizard:  validate_vm_ha_state: Validating VM HA-state
Dec 16 09:40:20 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] state [ false ] = OK
Dec 16 09:40:20 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] state [ false ] = OK
Dec 16 09:40:20 xen-01 ha-lizard:  Calling function write_pool_state
Dec 16 09:40:20 xen-01 ha-lizard: 8815 Calling function autoselect_slave
Dec 16 09:40:20 xen-01 ha-lizard: 8821 Calling function check_slave_status
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_master_mgt_link_state: Checking management interface link state
Dec 16 09:40:20 xen-01 ha-lizard: 8815 autoselect_slave: This host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:20 xen-01 ha-lizard: 8815 autoselect_slave: MASTER host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:20 xen-01 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_slave_status: Management link OK - continue
Dec 16 09:40:20 xen-01 ha-lizard: 8815 autoselect_slave: 28dfaba4-5b90-4535-afec-a244a6c3f88a is Master UUID - excluding from list of available slaves
Dec 16 09:40:20 xen-01 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:40:20 xen-01 ha-lizard: 8815 autoselect_slave: Removing Slave UUID from list of Hosts - Slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce is disabled or in maintenance mode
Dec 16 09:40:20 xen-01 ha-lizard: 8821 get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:20 xen-01 ha-lizard: 8815 autoselect_slave: 0 available Slave UUIDs found:
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_slave_status: Removing Master UUID from list of Hosts
Dec 16 09:40:20 xen-01 ha-lizard: 8815 autoselect_slave: New autopromote slave selected - updating pool-param:autopromote_uuid with
Dec 16 09:40:20 xen-01 ha-lizard: 8815 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_slave_status: Removing Slave UUID from list of Hosts - Slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce is disabled or in maintenance mode
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_slave_status: Host IP Address check Status Array for Slaves = ()
Dec 16 09:40:20 xen-01 ha-lizard: 8815 check_email_enabled: Email enabled for autoselect_slave
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_slave_status: Quorum check called
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_quorum: Checking host IPs: 172.16.14.199 172.16.14.200
Dec 16 09:40:20 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_quorum: Host IP: 172.16.14.199 Response = OK
Dec 16 09:40:20 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_quorum: LIVE HOSTs = 1
Dec 16 09:40:20 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_quorum: Host IP: 172.16.14.200 Response = OK
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_quorum: LIVE HOSTs = 2
Dec 16 09:40:20 xen-01 ha-lizard: 8815 email: Sending ALERT email to root@localhost: autoselect_slave:  NOTICE: New autopromote slave selected - updating pool-param:autopromote_uuid with
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_quorum: Using network points: 172.16.14.165 as possible additional vote
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_quorum: Heuristic IP: 172.16.14.165 Response = OK
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_quorum: Successful Replies = 1
Dec 16 09:40:20 xen-01 ha-lizard: 8821 Total enpoints checked = 1 with total successful replies = 1
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_quorum: Additional heuristic vote success. Incremeting vote by 1
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_slave_status: Failed slave count = 0
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_slave_status: No Failed slaves detected
Dec 16 09:40:20 xen-01 ha-lizard:  check_ha_enabled: Host is in maintenance mode
Dec 16 09:40:20 xen-01 ha-lizard:  check_ha_enabled: Master cannot be in maintenance mode - enabling host
Dec 16 09:40:20 xen-01 ha-lizard: 8821 Function check_slave_status reported no failures: calling vm_mon
Dec 16 09:40:20 xen-01 ha-lizard: 8821 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Dec 16 09:40:20 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: The host toolstack is still initialising. Please wait.
Dec 16 09:40:20 xen-01 ha-lizard:  check_ha_enabled: Host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ] has been enabled
Dec 16 09:40:20 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Traceback (most recent call last):
Dec 16 09:40:20 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/etc/ha-lizard/scripts/email_alert.py", line 105, in <module>
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email from: root@localhost
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email to: root@localhost
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xen-01
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Timestamp: Fri Dec 16 09:40:20 GMT 2016
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Process: ha-lizard Version: 2.1.3
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Content: autoselect_slave: NOTICE: New autopromote slave selected - updating pool-param:autopromote_uuid with
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: 
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool UUID       : e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Name-Label : XenPool
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master UUID: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:20 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     message = smtplib.SMTP(smtp_server, smtp_port, hostname)
Dec 16 09:40:20 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-01
Dec 16 09:40:20 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     (code, msg) = self.connect(host, port)
Dec 16 09:40:20 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 315, in connect
Dec 16 09:40:20 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     self.sock = self._get_socket(host, port, self.timeout)
Dec 16 09:40:20 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket
Dec 16 09:40:20 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     return socket.create_connection((host, port), timeout)
Dec 16 09:40:20 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : true
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-02
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : false
Dec 16 09:40:20 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Hostname: xen-01
Dec 16 09:40:20 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     raise err
Dec 16 09:40:20 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: socket.error: [Errno 111] Connection refused
Dec 16 09:40:20 xen-01 ha-lizard: 8815 email: Message copied to /dev/shm/ha-lizard-mail/1481881220.msg - Suppressing duplicates for 60 Minutes
Dec 16 09:40:20 xen-01 ha-lizard: 8821 vm_mon: Retrived list of VMs for this poll: 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:20 xen-01 ha-lizard: 8821 vm_mon: Removing Control Domains from VM list
Dec 16 09:40:20 xen-01 ha-lizard: 8821 vm_mon: VM list returned = 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:20 xen-01 ha-lizard: 8821 vm_state: Machine state for 4c4ca253-ff4f-2702-94a2-836fc0380f31 returned: running
Dec 16 09:40:20 xen-01 ha-lizard: 8821 vm_mon: VM 4c4ca253-ff4f-2702-94a2-836fc0380f31 state = running
Dec 16 09:40:20 xen-01 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:40:20 xen-01 ha-lizard: 8821 vm_state: Machine state for 174112ca-28d4-bbfe-6036-a2ee7a34e444 returned: halted
Dec 16 09:40:20 xen-01 ha-lizard: 8821 vm_mon: VM 174112ca-28d4-bbfe-6036-a2ee7a34e444 state = halted
Dec 16 09:40:20 xen-01 ha-lizard:  get_pool_host_list: found no host machines in pool - major error
Dec 16 09:40:20 xen-01 ha-lizard: 8821 vm_mon: GLOBAL_VM_HA is enabled. Adding VM: 174112ca-28d4-bbfe-6036-a2ee7a34e444 to list of failed VMs on this run.
Dec 16 09:40:20 xen-01 ha-lizard: 8821 vm_mon: 1 Eligible Halted VMs found
Dec 16 09:40:20 xen-01 ha-lizard: 8821 vm_mon: Halted VMs found: 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:20 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:20 xen-01 ha-lizard: 8821 vm_mon: Attempting to start VMs in halted state
Dec 16 09:40:20 xen-01 ha-lizard:  check_email_enabled: Email enabled for get_pool_host_list
Dec 16 09:40:20 xen-01 ha-lizard:  email: Sending ALERT email to root@localhost: get_pool_host_list:  Failed to get list of hosts in this pool
Dec 16 09:40:20 xen-01 ha-lizard: 8821 vm_mon: Starting VM: CentOS 7 (2) UUID: 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:20 xen-01 ha-lizard: 8821 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:20 xen-01 ha-lizard: 8821 check_email_enabled: Email enabled for vm_mon
Dec 16 09:40:21 xen-01 ha-lizard: 8821 email: Sending ALERT email to root@localhost: vm_mon:  Starting VM: CentOS 7 (2) UUID: 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email from: root@localhost
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email to: root@localhost
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xen-01
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Timestamp: Fri Dec 16 09:40:20 GMT 2016
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Process: ha-lizard Version: 2.1.3
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Content: get_pool_host_list: Failed to get list of hosts in this pool
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Traceback (most recent call last):
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: 
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool UUID       : e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/etc/ha-lizard/scripts/email_alert.py", line 105, in <module>
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Name-Label : XenPool
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master UUID: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     message = smtplib.SMTP(smtp_server, smtp_port, hostname)
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-01
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     (code, msg) = self.connect(host, port)
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 315, in connect
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     self.sock = self._get_socket(host, port, self.timeout)
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     return socket.create_connection((host, port), timeout)
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : true
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-02
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : false
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Hostname: xen-01
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     raise err
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: socket.error: [Errno 111] Connection refused
Dec 16 09:40:21 xen-01 ha-lizard:  email: Message copied to /dev/shm/ha-lizard-mail/1481881221.msg - Suppressing duplicates for 60 Minutes
Dec 16 09:40:21 xen-01 ha-lizard:  write_status_report: Writing status report
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Traceback (most recent call last):
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/etc/ha-lizard/scripts/email_alert.py", line 105, in <module>
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email from: root@localhost
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     message = smtplib.SMTP(smtp_server, smtp_port, hostname)
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email to: root@localhost
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xen-01
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Timestamp: Fri Dec 16 09:40:21 GMT 2016
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Process: ha-lizard Version: 2.1.3
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Content: vm_mon: Starting VM: CentOS 7 (2) UUID: 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: 
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool UUID       : e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Name-Label : XenPool
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master UUID: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-01
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : true
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-02
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : false
Dec 16 09:40:21 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Hostname: xen-01
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     (code, msg) = self.connect(host, port)
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 315, in connect
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     self.sock = self._get_socket(host, port, self.timeout)
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     return socket.create_connection((host, port), timeout)
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     raise err
Dec 16 09:40:21 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: socket.error: [Errno 111] Connection refused
Dec 16 09:40:21 xen-01 ha-lizard: 8821 email: Message copied to /dev/shm/ha-lizard-mail/1481881221.msg - Suppressing duplicates for 60 Minutes
Dec 16 09:40:21 xen-01 ha-lizard: 8821 vm_mon: HOST_SELECT_METHOD set to [ 0 ] - checking for a healthy host
Dec 16 09:40:21 xen-01 ha-lizard: 8821 vm_mon: No host available to start VM
Dec 16 09:40:23 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:40:24 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:25 xen-01 iscsi-ha: 8391 Spawning new instance of iscsi-ha
Dec 16 09:40:25 xen-01 iscsi-ha: 8391 check_logger_processes Checking logger processes
Dec 16 09:40:25 xen-01 iscsi-ha: 8391 check_logger_processes No processes to clear
Dec 16 09:40:25 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:40:25 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:25 xen-01 iscsi-ha:  This iteration is count 6978
Dec 16 09:40:25 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:25 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:40:25 xen-01 iscsi-ha: 10070 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:40:25 xen-01 iscsi-ha: 10070 service_execute: System V mode detected
Dec 16 09:40:25 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 service_execute: Returning exit status [ 0 ]
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431736 nr:3152 dw:4369792 dr:1325208604 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 DRBD Resource: iscsi1 in Connected state
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 service_execute: systemctl mode being used
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 service_execute: Returning exit status [ 0 ]
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 CHECKING IP 127.0.0.1
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 CHECKING IP 172.16.14.199
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 CHECKING IP 172.16.14.47
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 CHECKING IP 10.10.10.1
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1 ]
Dec 16 09:40:26 xen-01 iscsi-ha: 10070 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:40:26 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:40:26 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:40:26 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:40:27 xen-01 iscsi-ha: 10070 check_ip_health: 10.10.10.3 response = FAIL
Dec 16 09:40:27 xen-01 iscsi-ha: 10070 email: Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: cat: /dev/shm/iscsi-ha-mail/*.msg: No such file or directory
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: /etc/iscsi-ha/iscsi-ha.func: line 282: [: -eq: unary operator expected
Dec 16 09:40:27 xen-01 iscsi-ha: 10070 email Sending ALERT email to root@localhost: check_ip_health: 10.10.10.3 response = FAIL
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: Traceback (most recent call last):
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sending email from: root@localhost
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sending email to: root@localhost
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:   File "/etc/iscsi-ha/scripts/email_alert.py", line 114, in <module>
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Email Alert Subject: HA-Lizard noSAN SYSTEM ALERT - FROM HOST: xen-01
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:     message = smtplib.SMTP(smtp_server, smtp_port, hostname)
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Email Alert Timestamp: Fri Dec 16 09:40:27 GMT 2016
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:   File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Email Alert Process: iscsi-ha Version: IHA_2.1.3
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Email Alert Message Content: check_ip_health: 10.10.10.3 response = FAIL
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Email Alert Message Hostname: xen-01
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:     (code, msg) = self.connect(host, port)
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:   File "/usr/lib64/python2.7/smtplib.py", line 315, in connect
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:     self.sock = self._get_socket(host, port, self.timeout)
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:   File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:     return socket.create_connection((host, port), timeout)
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:     raise err
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: socket.error: [Errno 111] Connection refused
Dec 16 09:40:27 xen-01 iscsi-ha: 10070 email: Message copied to /dev/shm/iscsi-ha-mail/1481881227.msg - Suppressing duplicates for 30 Minutes
Dec 16 09:40:27 xen-01 iscsi-ha: 10070 Virtual IP 10.10.10.3 expected local, not found. Initializing..
Dec 16 09:40:27 xen-01 iscsi-ha: 10070 mask_numbits: Mask 255.255.255.0 contains 24 bits.
Dec 16 09:40:27 xen-01 iscsi-ha: 10070 DRBD Virtual IP: 10.10.10.3 successfully added to local interface xenbr6
Dec 16 09:40:27 xen-01 iscsi-ha: 10070 email: Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: /etc/iscsi-ha/iscsi-ha.func: line 282: [: -eq: unary operator expected
Dec 16 09:40:27 xen-01 iscsi-ha: 10070 email Sending ALERT email to root@localhost: Storage Virtual IP: 10.10.10.3 successfully added to local interface xenbr6
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sending email from: root@localhost
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sending email to: root@localhost
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Email Alert Subject: HA-Lizard noSAN SYSTEM ALERT - FROM HOST: xen-01
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Email Alert Timestamp: Fri Dec 16 09:40:27 GMT 2016
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Email Alert Process: iscsi-ha Version: IHA_2.1.3
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: Traceback (most recent call last):
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Email Alert Message Content: Storage Virtual IP: 10.10.10.3 successfully added to local interface xenbr6
Dec 16 09:40:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Email Alert Message Hostname: xen-01
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:   File "/etc/iscsi-ha/scripts/email_alert.py", line 114, in <module>
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:     message = smtplib.SMTP(smtp_server, smtp_port, hostname)
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:   File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:     (code, msg) = self.connect(host, port)
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:   File "/usr/lib64/python2.7/smtplib.py", line 315, in connect
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:     self.sock = self._get_socket(host, port, self.timeout)
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:   File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:     return socket.create_connection((host, port), timeout)
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon:     raise err
Dec 16 09:40:27 xen-01 iscsi-ha-ERROR-/etc/iscsi-ha/init/iscsi-ha.mon: socket.error: [Errno 111] Connection refused
Dec 16 09:40:27 xen-01 iscsi-ha: 10070 email: Message copied to /dev/shm/iscsi-ha-mail/1481881227.msg - Suppressing duplicates for 30 Minutes
Dec 16 09:40:27 xen-01 iscsi-ha: 10070 Updating ARP for 10.10.10.3
Dec 16 09:40:28 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:28 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:29 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:33 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:40:34 xen-01 ha-lizard: 8626 Spawning new instance of ha-lizard
Dec 16 09:40:34 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:34 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:34 xen-01 ha-lizard:  This iteration is count 4634
Dec 16 09:40:34 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:34 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:40:34 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:40:34 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:34 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:40:34 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:40:34 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:40:34 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:40:34 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:40:34 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:40:34 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:40:34 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:40:34 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:40:34 xen-01 ha-lizard:  Master management link OK - checking prior link state
Dec 16 09:40:34 xen-01 ha-lizard:  This host detected as pool  Master
Dec 16 09:40:35 xen-01 ha-lizard:  Found 2 hosts in pool
Dec 16 09:40:35 xen-01 ha-lizard:  validate_vm_ha_state: Validating VM HA-state
Dec 16 09:40:35 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] state [ false ] = OK
Dec 16 09:40:35 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] state [ false ] = OK
Dec 16 09:40:35 xen-01 ha-lizard:  Calling function write_pool_state
Dec 16 09:40:35 xen-01 ha-lizard: 11719 Calling function autoselect_slave
Dec 16 09:40:35 xen-01 ha-lizard: 11724 Calling function check_slave_status
Dec 16 09:40:35 xen-01 ha-lizard: 11724 check_master_mgt_link_state: Checking management interface link state
Dec 16 09:40:35 xen-01 ha-lizard: 11719 autoselect_slave: This host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:35 xen-01 ha-lizard: 11719 autoselect_slave: MASTER host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:35 xen-01 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:35 xen-01 ha-lizard: 11724 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:40:35 xen-01 ha-lizard: 11719 autoselect_slave: 28dfaba4-5b90-4535-afec-a244a6c3f88a is Master UUID - excluding from list of available slaves
Dec 16 09:40:35 xen-01 ha-lizard: 11724 check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:40:35 xen-01 ha-lizard: 11724 check_slave_status: Management link OK - continue
Dec 16 09:40:35 xen-01 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:40:35 xen-01 ha-lizard: 11724 get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:35 xen-01 ha-lizard: 11719 autoselect_slave: 1 available Slave UUIDs found: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:35 xen-01 ha-lizard: 11724 check_slave_status: Removing Master UUID from list of Hosts
Dec 16 09:40:35 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:35 xen-01 ha-lizard: 11719 autoselect_slave: New autopromote slave selected - updating pool-param:autopromote_uuid with bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:35 xen-01 ha-lizard:  check_email_enabled: Email enabled for write_pool_state
Dec 16 09:40:35 xen-01 ha-lizard: 11719 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:35 xen-01 ha-lizard: 11719 check_email_enabled: Email enabled for autoselect_slave
Dec 16 09:40:35 xen-01 ha-lizard: 11724 get_pool_ip_list: returned 172.16.14.200
Dec 16 09:40:35 xen-01 ha-lizard:  email: Sending ALERT email to root@localhost: write_pool_state:  Error retrieving autopromote_uuid from pool configuration
Dec 16 09:40:35 xen-01 ha-lizard: 11719 email: Sending ALERT email to root@localhost: autoselect_slave:  NOTICE: New autopromote slave selected - updating pool-param:autopromote_uuid with bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:35 xen-01 ha-lizard: 11724 check_xapi: Pool Host 172.16.14.200 xapi status = 13
Dec 16 09:40:35 xen-01 ha-lizard: 11724 check_slave_status: Host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] marked as healthy
Dec 16 09:40:35 xen-01 ha-lizard: 11724 check_slave_status: Host IP Address check Status Array for Slaves = (1)
Dec 16 09:40:35 xen-01 ha-lizard: 11724 check_slave_status: Quorum check called
Dec 16 09:40:35 xen-01 ha-lizard: 11724 check_quorum: No host IPs detected - Do Not Fence - exiting now
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email from: root@localhost
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Traceback (most recent call last):
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email to: root@localhost
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xen-01
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/etc/ha-lizard/scripts/email_alert.py", line 105, in <module>
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Timestamp: Fri Dec 16 09:40:35 GMT 2016
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     message = smtplib.SMTP(smtp_server, smtp_port, hostname)
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Process: ha-lizard Version: 2.1.3
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Content: write_pool_state: Error retrieving autopromote_uuid from pool configuration
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: 
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool UUID       : e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Name-Label : XenPool
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master UUID: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     (code, msg) = self.connect(host, port)
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 315, in connect
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     self.sock = self._get_socket(host, port, self.timeout)
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-01
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : true
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     return socket.create_connection((host, port), timeout)
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-02
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : false
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Hostname: xen-01
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     raise err
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: socket.error: [Errno 111] Connection refused
Dec 16 09:40:35 xen-01 ha-lizard:  email: Message copied to /dev/shm/ha-lizard-mail/1481881235.msg - Suppressing duplicates for 60 Minutes
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email from: root@localhost
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Traceback (most recent call last):
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email to: root@localhost
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/etc/ha-lizard/scripts/email_alert.py", line 105, in <module>
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xen-01
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Timestamp: Fri Dec 16 09:40:35 GMT 2016
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     message = smtplib.SMTP(smtp_server, smtp_port, hostname)
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Process: ha-lizard Version: 2.1.3
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Content: autoselect_slave: NOTICE: New autopromote slave selected - updating pool-param:autopromote_uuid with bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: 
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool UUID       : e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Name-Label : XenPool
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master UUID: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-01
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : true
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-02
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     (code, msg) = self.connect(host, port)
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 315, in connect
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : false
Dec 16 09:40:35 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Hostname: xen-01
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     self.sock = self._get_socket(host, port, self.timeout)
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     return socket.create_connection((host, port), timeout)
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     raise err
Dec 16 09:40:35 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: socket.error: [Errno 111] Connection refused
Dec 16 09:40:35 xen-01 ha-lizard: 11719 email: Message copied to /dev/shm/ha-lizard-mail/1481881235.msg - Suppressing duplicates for 60 Minutes
Dec 16 09:40:35 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:35 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:40:35 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:40:35 xen-01 iscsi-ha: 10044 Spawning new instance of iscsi-ha
Dec 16 09:40:35 xen-01 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:40:35 xen-01 iscsi-ha: 10044 check_logger_processes Checking logger processes
Dec 16 09:40:35 xen-01 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:36 xen-01 iscsi-ha: 10044 check_logger_processes No processes to clear
Dec 16 09:40:36 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:40:36 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:40:36 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:36 xen-01 iscsi-ha:  This iteration is count 6979
Dec 16 09:40:36 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:40:36 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:36 xen-01 ha-lizard:  write_status_report: Writing status report
Dec 16 09:40:36 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 service_execute: System V mode detected
Dec 16 09:40:36 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 service_execute: Returning exit status [ 0 ]
Dec 16 09:40:36 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:36 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431744 nr:3152 dw:4369800 dr:1325208636 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 DRBD Resource: iscsi1 in Connected state
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 service_execute: systemctl mode being used
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 service_execute: Returning exit status [ 0 ]
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 CHECKING IP 127.0.0.1
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 CHECKING IP 172.16.14.199
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 CHECKING IP 172.16.14.47
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 CHECKING IP 10.10.10.1
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 CHECKING IP 10.10.10.3
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:40:36 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:40:36 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:40:36 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:40:36 xen-01 iscsi-ha: 12140 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:40:36 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:40:36 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:40:36 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:40:39 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:43 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:40:44 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:46 xen-01 iscsi-ha: 12111 Spawning new instance of iscsi-ha
Dec 16 09:40:46 xen-01 iscsi-ha: 12111 check_logger_processes Checking logger processes
Dec 16 09:40:46 xen-01 iscsi-ha: 12111 check_logger_processes No processes to clear
Dec 16 09:40:46 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:40:46 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:46 xen-01 iscsi-ha:  This iteration is count 6980
Dec 16 09:40:46 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:46 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 service_execute: System V mode detected
Dec 16 09:40:46 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 service_execute: Returning exit status [ 0 ]
Dec 16 09:40:46 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:46 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431744 nr:3152 dw:4369800 dr:1325208652 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 DRBD Resource: iscsi1 in Connected state
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 service_execute: systemctl mode being used
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 service_execute: Returning exit status [ 0 ]
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 CHECKING IP 127.0.0.1
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 CHECKING IP 172.16.14.199
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 CHECKING IP 172.16.14.47
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 CHECKING IP 10.10.10.1
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 CHECKING IP 10.10.10.3
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:40:46 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:40:46 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:40:46 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:40:46 xen-01 iscsi-ha: 12458 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:40:46 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:40:46 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:40:46 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:40:49 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:49 xen-01 ha-lizard: 11543 Spawning new instance of ha-lizard
Dec 16 09:40:49 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:49 xen-01 ha-lizard:  This iteration is count 4635
Dec 16 09:40:49 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:49 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:40:49 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:40:49 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:49 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:40:49 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:40:49 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:40:49 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:40:49 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:40:49 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:40:50 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:40:50 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:40:50 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:40:50 xen-01 ha-lizard:  Master management link OK - checking prior link state
Dec 16 09:40:50 xen-01 ha-lizard:  This host detected as pool  Master
Dec 16 09:40:50 xen-01 ha-lizard:  Found 2 hosts in pool
Dec 16 09:40:50 xen-01 ha-lizard:  validate_vm_ha_state: Validating VM HA-state
Dec 16 09:40:50 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] state [ false ] = OK
Dec 16 09:40:50 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] state [ false ] = OK
Dec 16 09:40:50 xen-01 ha-lizard:  Calling function write_pool_state
Dec 16 09:40:50 xen-01 ha-lizard: 12994 Calling function autoselect_slave
Dec 16 09:40:50 xen-01 ha-lizard: 13001 Calling function check_slave_status
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_master_mgt_link_state: Checking management interface link state
Dec 16 09:40:50 xen-01 ha-lizard: 12994 autoselect_slave: This host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:50 xen-01 ha-lizard: 12994 autoselect_slave: MASTER host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:50 xen-01 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:40:50 xen-01 ha-lizard: 12994 autoselect_slave: 28dfaba4-5b90-4535-afec-a244a6c3f88a is Master UUID - excluding from list of available slaves
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_slave_status: Management link OK - continue
Dec 16 09:40:50 xen-01 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31
Dec 16 09:40:50 xen-01 ha-lizard: 12994 autoselect_slave: 1 available Slave UUIDs found: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:50 xen-01 ha-lizard: 13001 get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_slave_status: Removing Master UUID from list of Hosts
Dec 16 09:40:50 xen-01 ha-lizard: 12994 autoselect_slave: Selected Slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce = Current slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce - ignoring update
Dec 16 09:40:50 xen-01 ha-lizard: 13001 get_pool_ip_list: returned 172.16.14.200
Dec 16 09:40:50 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:50 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:40:50 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_xapi: Pool Host 172.16.14.200 xapi status = 13
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_slave_status: Host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] marked as healthy
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_slave_status: Host IP Address check Status Array for Slaves = (1)
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_slave_status: Quorum check called
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_quorum: Checking host IPs: 172.16.14.199 172.16.14.200
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_quorum: Host IP: 172.16.14.199 Response = OK
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_quorum: LIVE HOSTs = 1
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_quorum: Host IP: 172.16.14.200 Response = OK
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_quorum: LIVE HOSTs = 2
Dec 16 09:40:50 xen-01 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_quorum: Using network points: 172.16.14.165 as possible additional vote
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_quorum: Heuristic IP: 172.16.14.165 Response = OK
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_quorum: Successful Replies = 1
Dec 16 09:40:50 xen-01 ha-lizard: 13001 Total enpoints checked = 1 with total successful replies = 1
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_quorum: Additional heuristic vote success. Incremeting vote by 1
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Dec 16 09:40:50 xen-01 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_slave_status: Failed slave count = 0
Dec 16 09:40:50 xen-01 ha-lizard: 13001 check_slave_status: No Failed slaves detected
Dec 16 09:40:50 xen-01 ha-lizard: 13001 Function check_slave_status reported no failures: calling vm_mon
Dec 16 09:40:50 xen-01 ha-lizard: 13001 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Dec 16 09:40:50 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: Retrived list of VMs for this poll: 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: Removing Control Domains from VM list
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: VM list returned = 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:51 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:40:51 xen-01 ha-lizard:  write_status_report: Writing status report
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_state: Machine state for 4c4ca253-ff4f-2702-94a2-836fc0380f31 returned: running
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: VM 4c4ca253-ff4f-2702-94a2-836fc0380f31 state = running
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_state: Machine state for 174112ca-28d4-bbfe-6036-a2ee7a34e444 returned: halted
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: VM 174112ca-28d4-bbfe-6036-a2ee7a34e444 state = halted
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: GLOBAL_VM_HA is enabled. Adding VM: 174112ca-28d4-bbfe-6036-a2ee7a34e444 to list of failed VMs on this run.
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: 1 Eligible Halted VMs found
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: Halted VMs found: 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: Attempting to start VMs in halted state
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: Starting VM: CentOS 7 (2) UUID: 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:51 xen-01 ha-lizard: 13001 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:51 xen-01 ha-lizard: 13001 check_email_enabled: Email enabled for vm_mon
Dec 16 09:40:51 xen-01 ha-lizard: 13001 email: Duplicate message - not sending. Content = vm_mon:  Starting VM: CentOS 7 (2) UUID: 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:51 xen-01 ha-lizard: 13001 email: Message barred for 60 minutes
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: HOST_SELECT_METHOD set to [ 0 ] - checking for a healthy host
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: This host [ 28dfaba4-5b90-4535-afec-a244a6c3f88a ] start on serial [ 1829 ]
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: This host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] start on serial [ 575 ]
Dec 16 09:40:51 xen-01 ha-lizard: 13001 vm_mon: Host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] health status = [ healthy ]
Dec 16 09:40:53 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:40:54 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:55 xen-01 ha-lizard: 13001 vm_mon: VM start exit result = [ 1 ]
Dec 16 09:40:55 xen-01 ha-lizard: 13001 vm_mon: VM start returned  messages = [ Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=VDI 3d66fc17-d503-4bf4-aad8-7a4619a8425a not detached cleanly],  ]
Dec 16 09:40:55 xen-01 ha-lizard: 13001 vm_mon: Error code: SR_BACKEND_FAILURE_46#012Error parameters: , The VDI is not available [opterr=VDI 3d66fc17-d503-4bf4-aad8-7a4619a8425a not detached cleanly],
Dec 16 09:40:55 xen-01 ha-lizard: 13001 reset_vm_vdi: Resetting VDI(s) for VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ]
Dec 16 09:40:55 xen-01 ha-lizard: 13001 reset_vm_vdi: No VDI found for VBD [ 83c80601-1918-22bf-5cd3-cc21781c6c9e ]
Dec 16 09:40:55 xen-01 ha-lizard: 13001 reset_vm_vdi: Found VDI [ 3d66fc17-d503-4bf4-aad8-7a4619a8425a ]
Dec 16 09:40:55 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Force-cleared RW for 3d66fc17-d503-4bf4-aad8-7a4619a8425a on host 28dfaba4-5b90-4535-afec-a244a6c3f88a (xen-01)
Dec 16 09:40:55 xen-01 ha-lizard: 13001 reset_vm_vdi: VDI [ 3d66fc17-d503-4bf4-aad8-7a4619a8425a ] reset
Dec 16 09:40:55 xen-01 ha-lizard: 13001 vm_mon: Reattempting vm [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] start
Dec 16 09:40:56 xen-01 iscsi-ha: 12444 Spawning new instance of iscsi-ha
Dec 16 09:40:56 xen-01 iscsi-ha: 12444 check_logger_processes Checking logger processes
Dec 16 09:40:56 xen-01 iscsi-ha: 12444 check_logger_processes No processes to clear
Dec 16 09:40:56 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:40:56 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:40:56 xen-01 iscsi-ha:  This iteration is count 6981
Dec 16 09:40:56 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:40:56 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 service_execute: System V mode detected
Dec 16 09:40:56 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 service_execute: Returning exit status [ 0 ]
Dec 16 09:40:56 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:56 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431744 nr:3152 dw:4369800 dr:1325208668 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 DRBD Resource: iscsi1 in Connected state
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 service_execute: systemctl mode being used
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 service_execute: Returning exit status [ 0 ]
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 CHECKING IP 127.0.0.1
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 CHECKING IP 172.16.14.199
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 CHECKING IP 172.16.14.47
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 CHECKING IP 10.10.10.1
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 CHECKING IP 10.10.10.3
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:40:56 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:40:56 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:40:56 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:40:56 xen-01 iscsi-ha: 13639 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:40:56 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:40:56 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:40:56 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:40:59 xen-01 ha-lizard: 13001 vm_mon: Successfully started VM: CentOS 7 (2) UUID: 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:59 xen-01 ha-lizard: 13001 Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:40:59 xen-01 ha-lizard: 13001 check_email_enabled: Email enabled for vm_mon
Dec 16 09:40:59 xen-01 ha-lizard: 13001 email: Sending ALERT email to root@localhost: vm_mon:  Successfully started VM: CentOS 7 (2) UUID: 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:59 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email from: root@localhost
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Sending email to: root@localhost
Dec 16 09:40:59 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Traceback (most recent call last):
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Subject: SYSTEM_ALERT-FROM_HOST:xen-01
Dec 16 09:40:59 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/etc/ha-lizard/scripts/email_alert.py", line 105, in <module>
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Timestamp: Fri Dec 16 09:40:59 GMT 2016
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Process: ha-lizard Version: 2.1.3
Dec 16 09:40:59 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     message = smtplib.SMTP(smtp_server, smtp_port, hostname)
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Content: vm_mon: Successfully started VM: CentOS 7 (2) UUID: 174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:40:59 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 255, in __init__
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: 
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool UUID       : e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Name-Label : XenPool
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master UUID: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:59 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     (code, msg) = self.connect(host, port)
Dec 16 09:40:59 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 315, in connect
Dec 16 09:40:59 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     self.sock = self._get_socket(host, port, self.timeout)
Dec 16 09:40:59 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/smtplib.py", line 290, in _get_socket
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-01
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : true
Dec 16 09:40:59 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     return socket.create_connection((host, port), timeout)
Dec 16 09:40:59 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:   File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: --------------------------------------------------------
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host Name-Label : xen-02
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Host UUID       : bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Pool Master     : false
Dec 16 09:40:59 xen-01 ha-lizard-NOTICE-/etc/ha-lizard/init/ha-lizard.mon: Email Alert Message Hostname: xen-01
Dec 16 09:40:59 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon:     raise err
Dec 16 09:40:59 xen-01 ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: socket.error: [Errno 111] Connection refused
Dec 16 09:40:59 xen-01 ha-lizard: 13001 email: Message copied to /dev/shm/ha-lizard-mail/1481881259.msg - Suppressing duplicates for 60 Minutes
Dec 16 09:41:03 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:41:04 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:04 xen-01 ha-lizard: 12820 Spawning new instance of ha-lizard
Dec 16 09:41:04 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:41:04 xen-01 ha-lizard:  This iteration is count 4636
Dec 16 09:41:04 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:04 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:41:04 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:41:04 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:04 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:04 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:04 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:41:04 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:41:05 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:41:05 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:41:05 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:41:05 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:41:05 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:41:05 xen-01 ha-lizard:  Master management link OK - checking prior link state
Dec 16 09:41:05 xen-01 ha-lizard:  This host detected as pool  Master
Dec 16 09:41:05 xen-01 ha-lizard:  Found 2 hosts in pool
Dec 16 09:41:05 xen-01 ha-lizard:  validate_vm_ha_state: Validating VM HA-state
Dec 16 09:41:05 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] state [ false ] = OK
Dec 16 09:41:05 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] state [ false ] = OK
Dec 16 09:41:05 xen-01 ha-lizard:  Calling function write_pool_state
Dec 16 09:41:05 xen-01 ha-lizard: 14489 Calling function autoselect_slave
Dec 16 09:41:05 xen-01 ha-lizard: 14495 Calling function check_slave_status
Dec 16 09:41:05 xen-01 ha-lizard: 14495 check_master_mgt_link_state: Checking management interface link state
Dec 16 09:41:05 xen-01 ha-lizard: 14489 autoselect_slave: This host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:05 xen-01 ha-lizard: 14489 autoselect_slave: MASTER host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:05 xen-01 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:05 xen-01 ha-lizard: 14495 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:41:05 xen-01 ha-lizard: 14489 autoselect_slave: 28dfaba4-5b90-4535-afec-a244a6c3f88a is Master UUID - excluding from list of available slaves
Dec 16 09:41:05 xen-01 ha-lizard: 14495 check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:41:05 xen-01 ha-lizard: 14495 check_slave_status: Management link OK - continue
Dec 16 09:41:05 xen-01 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:05 xen-01 ha-lizard: 14489 autoselect_slave: 1 available Slave UUIDs found: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:05 xen-01 ha-lizard: 14495 get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:05 xen-01 ha-lizard: 14495 check_slave_status: Removing Master UUID from list of Hosts
Dec 16 09:41:05 xen-01 ha-lizard: 14489 autoselect_slave: Selected Slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce = Current slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce - ignoring update
Dec 16 09:41:05 xen-01 ha-lizard: 14495 get_pool_ip_list: returned 172.16.14.200
Dec 16 09:41:05 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:05 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:05 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:05 xen-01 ha-lizard: 14495 check_xapi: Pool Host 172.16.14.200 xapi status = 13
Dec 16 09:41:05 xen-01 ha-lizard: 14495 check_slave_status: Host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] marked as healthy
Dec 16 09:41:05 xen-01 ha-lizard: 14495 check_slave_status: Host IP Address check Status Array for Slaves = (1)
Dec 16 09:41:05 xen-01 ha-lizard: 14495 check_slave_status: Quorum check called
Dec 16 09:41:06 xen-01 ha-lizard: 14495 check_quorum: Checking host IPs: 172.16.14.199 172.16.14.200
Dec 16 09:41:06 xen-01 ha-lizard: 14495 check_quorum: Host IP: 172.16.14.199 Response = OK
Dec 16 09:41:06 xen-01 ha-lizard: 14495 check_quorum: LIVE HOSTs = 1
Dec 16 09:41:06 xen-01 ha-lizard: 14495 check_quorum: Host IP: 172.16.14.200 Response = OK
Dec 16 09:41:06 xen-01 ha-lizard: 14495 check_quorum: LIVE HOSTs = 2
Dec 16 09:41:06 xen-01 ha-lizard: 14495 check_quorum: Using network points: 172.16.14.165 as possible additional vote
Dec 16 09:41:06 xen-01 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:41:06 xen-01 ha-lizard: 14495 check_quorum: Heuristic IP: 172.16.14.165 Response = OK
Dec 16 09:41:06 xen-01 ha-lizard: 14495 check_quorum: Successful Replies = 1
Dec 16 09:41:06 xen-01 ha-lizard: 14495 Total enpoints checked = 1 with total successful replies = 1
Dec 16 09:41:06 xen-01 ha-lizard: 14495 check_quorum: Additional heuristic vote success. Incremeting vote by 1
Dec 16 09:41:06 xen-01 ha-lizard: 14495 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Dec 16 09:41:06 xen-01 ha-lizard: 14495 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed
Dec 16 09:41:06 xen-01 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:06 xen-01 ha-lizard: 14495 check_slave_status: Failed slave count = 0
Dec 16 09:41:06 xen-01 ha-lizard: 14495 check_slave_status: No Failed slaves detected
Dec 16 09:41:06 xen-01 ha-lizard: 14495 Function check_slave_status reported no failures: calling vm_mon
Dec 16 09:41:06 xen-01 ha-lizard: 14495 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Dec 16 09:41:06 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:41:06 xen-01 ha-lizard: 14495 vm_mon: Retrived list of VMs for this poll: 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:06 xen-01 ha-lizard: 14495 vm_mon: Removing Control Domains from VM list
Dec 16 09:41:06 xen-01 ha-lizard: 14495 vm_mon: VM list returned = 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:06 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:41:06 xen-01 ha-lizard:  write_status_report: Writing status report
Dec 16 09:41:06 xen-01 ha-lizard: 14495 vm_state: Machine state for 4c4ca253-ff4f-2702-94a2-836fc0380f31 returned: running
Dec 16 09:41:06 xen-01 ha-lizard: 14495 vm_mon: VM 4c4ca253-ff4f-2702-94a2-836fc0380f31 state = running
Dec 16 09:41:06 xen-01 ha-lizard: 14495 vm_state: Machine state for 174112ca-28d4-bbfe-6036-a2ee7a34e444 returned: running
Dec 16 09:41:06 xen-01 ha-lizard: 14495 vm_mon: VM 174112ca-28d4-bbfe-6036-a2ee7a34e444 state = running
Dec 16 09:41:06 xen-01 ha-lizard: 14495 vm_mon: 0 Eligible Halted VMs found
Dec 16 09:41:06 xen-01 iscsi-ha: 13625 Spawning new instance of iscsi-ha
Dec 16 09:41:06 xen-01 iscsi-ha: 13625 check_logger_processes Checking logger processes
Dec 16 09:41:06 xen-01 iscsi-ha: 13625 check_logger_processes No processes to clear
Dec 16 09:41:06 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:41:06 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:41:06 xen-01 iscsi-ha:  This iteration is count 6982
Dec 16 09:41:06 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:06 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 service_execute: System V mode detected
Dec 16 09:41:06 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 service_execute: Returning exit status [ 0 ]
Dec 16 09:41:06 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:06 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431748 nr:3152 dw:4369804 dr:1325227536 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 DRBD Resource: iscsi1 in Connected state
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 service_execute: systemctl mode being used
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 service_execute: Returning exit status [ 0 ]
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 CHECKING IP 127.0.0.1
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 CHECKING IP 172.16.14.199
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 CHECKING IP 172.16.14.47
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 CHECKING IP 10.10.10.1
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 CHECKING IP 10.10.10.3
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:41:06 xen-01 iscsi-ha: 15165 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:41:06 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:41:06 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:41:06 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:41:07 xen-01 iscsi-ha: 15165 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:41:07 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:41:07 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:41:07 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:41:09 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:13 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:41:14 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:16 xen-01 iscsi-ha: 15132 Spawning new instance of iscsi-ha
Dec 16 09:41:16 xen-01 iscsi-ha: 15132 check_logger_processes Checking logger processes
Dec 16 09:41:16 xen-01 iscsi-ha: 15132 check_logger_processes No processes to clear
Dec 16 09:41:16 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:41:16 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:41:16 xen-01 iscsi-ha:  This iteration is count 6983
Dec 16 09:41:16 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:16 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:41:16 xen-01 iscsi-ha: 15812 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:41:16 xen-01 iscsi-ha: 15812 service_execute: System V mode detected
Dec 16 09:41:16 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:16 xen-01 iscsi-ha: 15812 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:41:16 xen-01 iscsi-ha: 15812 service_execute: Returning exit status [ 0 ]
Dec 16 09:41:16 xen-01 iscsi-ha: 15812 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431748 nr:3152 dw:4369804 dr:1325279648 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:41:16 xen-01 iscsi-ha: 15812 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:41:16 xen-01 iscsi-ha: 15812 DRBD Resource: iscsi1 in Connected state
Dec 16 09:41:16 xen-01 iscsi-ha: 15812 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:41:16 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:16 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:16 xen-01 iscsi-ha: 15812 service_execute: systemctl mode being used
Dec 16 09:41:16 xen-01 iscsi-ha: 15812 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:41:16 xen-01 iscsi-ha: 15812 service_execute: Returning exit status [ 0 ]
Dec 16 09:41:16 xen-01 iscsi-ha: 15812 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:41:17 xen-01 iscsi-ha: 15812 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:41:17 xen-01 iscsi-ha: 15812 CHECKING IP 127.0.0.1
Dec 16 09:41:17 xen-01 iscsi-ha: 15812 CHECKING IP 172.16.14.199
Dec 16 09:41:17 xen-01 iscsi-ha: 15812 CHECKING IP 172.16.14.47
Dec 16 09:41:17 xen-01 iscsi-ha: 15812 CHECKING IP 10.10.10.1
Dec 16 09:41:17 xen-01 iscsi-ha: 15812 CHECKING IP 10.10.10.3
Dec 16 09:41:17 xen-01 iscsi-ha: 15812 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:41:17 xen-01 iscsi-ha: 15812 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:41:17 xen-01 iscsi-ha: 15812 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:41:17 xen-01 iscsi-ha: 15812 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:41:17 xen-01 iscsi-ha: 15812 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:41:17 xen-01 iscsi-ha: 15812 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:41:17 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:41:17 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:41:17 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:41:17 xen-01 iscsi-ha: 15812 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:41:17 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:41:17 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:41:17 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:41:19 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:19 xen-01 ha-lizard: 14171 Spawning new instance of ha-lizard
Dec 16 09:41:19 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:41:19 xen-01 ha-lizard:  This iteration is count 4637
Dec 16 09:41:19 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:19 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:41:19 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:41:20 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:20 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:20 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:20 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:41:20 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:41:20 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:41:20 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:41:20 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:41:20 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:41:20 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:41:20 xen-01 ha-lizard:  Master management link OK - checking prior link state
Dec 16 09:41:20 xen-01 ha-lizard:  This host detected as pool  Master
Dec 16 09:41:20 xen-01 ha-lizard:  Found 2 hosts in pool
Dec 16 09:41:20 xen-01 ha-lizard:  validate_vm_ha_state: Validating VM HA-state
Dec 16 09:41:20 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] state [ false ] = OK
Dec 16 09:41:20 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] state [ false ] = OK
Dec 16 09:41:20 xen-01 ha-lizard:  Calling function write_pool_state
Dec 16 09:41:20 xen-01 ha-lizard: 16345 Calling function autoselect_slave
Dec 16 09:41:20 xen-01 ha-lizard: 16350 Calling function check_slave_status
Dec 16 09:41:20 xen-01 ha-lizard: 16350 check_master_mgt_link_state: Checking management interface link state
Dec 16 09:41:20 xen-01 ha-lizard: 16345 autoselect_slave: This host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:20 xen-01 ha-lizard: 16345 autoselect_slave: MASTER host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:20 xen-01 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:20 xen-01 ha-lizard: 16350 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:41:20 xen-01 ha-lizard: 16345 autoselect_slave: 28dfaba4-5b90-4535-afec-a244a6c3f88a is Master UUID - excluding from list of available slaves
Dec 16 09:41:20 xen-01 ha-lizard: 16350 check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:41:20 xen-01 ha-lizard: 16350 check_slave_status: Management link OK - continue
Dec 16 09:41:20 xen-01 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:20 xen-01 ha-lizard: 16345 autoselect_slave: 1 available Slave UUIDs found: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:20 xen-01 ha-lizard: 16350 get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:20 xen-01 ha-lizard: 16350 check_slave_status: Removing Master UUID from list of Hosts
Dec 16 09:41:20 xen-01 ha-lizard: 16345 autoselect_slave: Selected Slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce = Current slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce - ignoring update
Dec 16 09:41:20 xen-01 ha-lizard: 16350 get_pool_ip_list: returned 172.16.14.200
Dec 16 09:41:20 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:20 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:20 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_xapi: Pool Host 172.16.14.200 xapi status = 13
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_slave_status: Host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] marked as healthy
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_slave_status: Host IP Address check Status Array for Slaves = (1)
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_slave_status: Quorum check called
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_quorum: Checking host IPs: 172.16.14.199 172.16.14.200
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_quorum: Host IP: 172.16.14.199 Response = OK
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_quorum: LIVE HOSTs = 1
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_quorum: Host IP: 172.16.14.200 Response = OK
Dec 16 09:41:21 xen-01 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_quorum: LIVE HOSTs = 2
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_quorum: Using network points: 172.16.14.165 as possible additional vote
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_quorum: Heuristic IP: 172.16.14.165 Response = OK
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_quorum: Successful Replies = 1
Dec 16 09:41:21 xen-01 ha-lizard: 16350 Total enpoints checked = 1 with total successful replies = 1
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_quorum: Additional heuristic vote success. Incremeting vote by 1
Dec 16 09:41:21 xen-01 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_slave_status: Failed slave count = 0
Dec 16 09:41:21 xen-01 ha-lizard: 16350 check_slave_status: No Failed slaves detected
Dec 16 09:41:21 xen-01 ha-lizard: 16350 Function check_slave_status reported no failures: calling vm_mon
Dec 16 09:41:21 xen-01 ha-lizard: 16350 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Dec 16 09:41:21 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:41:21 xen-01 ha-lizard: 16350 vm_mon: Retrived list of VMs for this poll: 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:21 xen-01 ha-lizard: 16350 vm_mon: Removing Control Domains from VM list
Dec 16 09:41:21 xen-01 ha-lizard: 16350 vm_mon: VM list returned = 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:21 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:41:21 xen-01 ha-lizard:  write_status_report: Writing status report
Dec 16 09:41:21 xen-01 ha-lizard: 16350 vm_state: Machine state for 4c4ca253-ff4f-2702-94a2-836fc0380f31 returned: running
Dec 16 09:41:21 xen-01 ha-lizard: 16350 vm_mon: VM 4c4ca253-ff4f-2702-94a2-836fc0380f31 state = running
Dec 16 09:41:21 xen-01 ha-lizard: 16350 vm_state: Machine state for 174112ca-28d4-bbfe-6036-a2ee7a34e444 returned: running
Dec 16 09:41:21 xen-01 ha-lizard: 16350 vm_mon: VM 174112ca-28d4-bbfe-6036-a2ee7a34e444 state = running
Dec 16 09:41:21 xen-01 ha-lizard: 16350 vm_mon: 0 Eligible Halted VMs found
Dec 16 09:41:23 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:41:24 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:25 xen-01 xsconsole: Data update connection failed - retrying.  Exception was:
Dec 16 09:41:25 xen-01 xsconsole: ['HANDLE_INVALID', 'session', 'OpaqueRef:79b78e87-fdb4-873d-e737-eb3d2166a388']
Dec 16 09:41:26 xen-01 iscsi-ha: 15796 Spawning new instance of iscsi-ha
Dec 16 09:41:26 xen-01 iscsi-ha: 15796 check_logger_processes Checking logger processes
Dec 16 09:41:26 xen-01 iscsi-ha: 15796 check_logger_processes No processes to clear
Dec 16 09:41:26 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:41:26 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:41:26 xen-01 iscsi-ha:  This iteration is count 6984
Dec 16 09:41:26 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:26 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:41:26 xen-01 iscsi-ha: 16988 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:41:26 xen-01 iscsi-ha: 16988 service_execute: System V mode detected
Dec 16 09:41:26 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 service_execute: Returning exit status [ 0 ]
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431748 nr:3152 dw:4369804 dr:1325355832 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:41:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 DRBD Resource: iscsi1 in Connected state
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 service_execute: systemctl mode being used
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 service_execute: Returning exit status [ 0 ]
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 CHECKING IP 127.0.0.1
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 CHECKING IP 172.16.14.199
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 CHECKING IP 172.16.14.47
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 CHECKING IP 10.10.10.1
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 CHECKING IP 10.10.10.3
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:41:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:41:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:41:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:41:27 xen-01 iscsi-ha: 16988 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:41:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:41:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:41:27 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:41:29 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:33 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:41:34 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:34 xen-01 ha-lizard: 16171 Spawning new instance of ha-lizard
Dec 16 09:41:35 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:41:35 xen-01 ha-lizard:  This iteration is count 4638
Dec 16 09:41:35 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:35 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:41:35 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:41:35 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:35 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:35 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:35 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:41:35 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:41:35 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:41:35 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:41:35 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:41:35 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:41:35 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:41:35 xen-01 ha-lizard:  Master management link OK - checking prior link state
Dec 16 09:41:35 xen-01 ha-lizard:  This host detected as pool  Master
Dec 16 09:41:35 xen-01 ha-lizard:  Found 2 hosts in pool
Dec 16 09:41:35 xen-01 ha-lizard:  validate_vm_ha_state: Validating VM HA-state
Dec 16 09:41:35 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] state [ false ] = OK
Dec 16 09:41:35 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] state [ false ] = OK
Dec 16 09:41:35 xen-01 ha-lizard:  Calling function write_pool_state
Dec 16 09:41:35 xen-01 ha-lizard: 17437 Calling function autoselect_slave
Dec 16 09:41:35 xen-01 ha-lizard: 17443 Calling function check_slave_status
Dec 16 09:41:35 xen-01 ha-lizard: 17443 check_master_mgt_link_state: Checking management interface link state
Dec 16 09:41:35 xen-01 ha-lizard: 17437 autoselect_slave: This host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:35 xen-01 ha-lizard: 17437 autoselect_slave: MASTER host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:35 xen-01 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:35 xen-01 ha-lizard: 17443 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:41:35 xen-01 ha-lizard: 17437 autoselect_slave: 28dfaba4-5b90-4535-afec-a244a6c3f88a is Master UUID - excluding from list of available slaves
Dec 16 09:41:35 xen-01 ha-lizard: 17443 check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:41:35 xen-01 ha-lizard: 17443 check_slave_status: Management link OK - continue
Dec 16 09:41:36 xen-01 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:36 xen-01 ha-lizard: 17437 autoselect_slave: 1 available Slave UUIDs found: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:36 xen-01 ha-lizard: 17443 get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_slave_status: Removing Master UUID from list of Hosts
Dec 16 09:41:36 xen-01 ha-lizard: 17437 autoselect_slave: Selected Slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce = Current slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce - ignoring update
Dec 16 09:41:36 xen-01 ha-lizard: 17443 get_pool_ip_list: returned 172.16.14.200
Dec 16 09:41:36 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:36 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:36 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_xapi: Pool Host 172.16.14.200 xapi status = 13
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_slave_status: Host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] marked as healthy
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_slave_status: Host IP Address check Status Array for Slaves = (1)
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_slave_status: Quorum check called
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_quorum: Checking host IPs: 172.16.14.199 172.16.14.200
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_quorum: Host IP: 172.16.14.199 Response = OK
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_quorum: LIVE HOSTs = 1
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_quorum: Host IP: 172.16.14.200 Response = OK
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_quorum: LIVE HOSTs = 2
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_quorum: Using network points: 172.16.14.165 as possible additional vote
Dec 16 09:41:36 xen-01 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_quorum: Heuristic IP: 172.16.14.165 Response = OK
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_quorum: Successful Replies = 1
Dec 16 09:41:36 xen-01 ha-lizard: 17443 Total enpoints checked = 1 with total successful replies = 1
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_quorum: Additional heuristic vote success. Incremeting vote by 1
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_slave_status: Failed slave count = 0
Dec 16 09:41:36 xen-01 ha-lizard: 17443 check_slave_status: No Failed slaves detected
Dec 16 09:41:36 xen-01 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:36 xen-01 ha-lizard: 17443 Function check_slave_status reported no failures: calling vm_mon
Dec 16 09:41:36 xen-01 ha-lizard: 17443 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Dec 16 09:41:36 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:41:36 xen-01 ha-lizard: 17443 vm_mon: Retrived list of VMs for this poll: 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:36 xen-01 ha-lizard: 17443 vm_mon: Removing Control Domains from VM list
Dec 16 09:41:36 xen-01 ha-lizard: 17443 vm_mon: VM list returned = 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:36 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:41:36 xen-01 ha-lizard:  write_status_report: Writing status report
Dec 16 09:41:36 xen-01 ha-lizard: 17443 vm_state: Machine state for 4c4ca253-ff4f-2702-94a2-836fc0380f31 returned: running
Dec 16 09:41:36 xen-01 ha-lizard: 17443 vm_mon: VM 4c4ca253-ff4f-2702-94a2-836fc0380f31 state = running
Dec 16 09:41:36 xen-01 ha-lizard: 17443 vm_state: Machine state for 174112ca-28d4-bbfe-6036-a2ee7a34e444 returned: running
Dec 16 09:41:36 xen-01 ha-lizard: 17443 vm_mon: VM 174112ca-28d4-bbfe-6036-a2ee7a34e444 state = running
Dec 16 09:41:36 xen-01 ha-lizard: 17443 vm_mon: 0 Eligible Halted VMs found
Dec 16 09:41:36 xen-01 iscsi-ha: 16967 Spawning new instance of iscsi-ha
Dec 16 09:41:36 xen-01 iscsi-ha: 16967 check_logger_processes Checking logger processes
Dec 16 09:41:37 xen-01 iscsi-ha: 16967 check_logger_processes No processes to clear
Dec 16 09:41:37 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:41:37 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:41:37 xen-01 iscsi-ha:  This iteration is count 6985
Dec 16 09:41:37 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:37 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 service_execute: System V mode detected
Dec 16 09:41:37 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 service_execute: Returning exit status [ 0 ]
Dec 16 09:41:37 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:37 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329431748 nr:3152 dw:4369804 dr:1325355848 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 DRBD Resource: iscsi1 in Connected state
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 service_execute: systemctl mode being used
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 service_execute: Returning exit status [ 0 ]
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 CHECKING IP 127.0.0.1
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 CHECKING IP 172.16.14.199
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 CHECKING IP 172.16.14.47
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 CHECKING IP 10.10.10.1
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 CHECKING IP 10.10.10.3
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:41:37 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:41:37 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:41:37 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:41:37 xen-01 iscsi-ha: 17904 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:41:37 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:41:37 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:41:37 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:41:39 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:43 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:41:44 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:47 xen-01 iscsi-ha: 17890 Spawning new instance of iscsi-ha
Dec 16 09:41:47 xen-01 iscsi-ha: 17890 check_logger_processes Checking logger processes
Dec 16 09:41:47 xen-01 iscsi-ha: 17890 check_logger_processes No processes to clear
Dec 16 09:41:47 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:41:47 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:41:47 xen-01 iscsi-ha:  This iteration is count 6986
Dec 16 09:41:47 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:47 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 service_execute: System V mode detected
Dec 16 09:41:47 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 service_execute: Returning exit status [ 0 ]
Dec 16 09:41:47 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:47 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329436024 nr:3152 dw:4374080 dr:1325355864 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 DRBD Resource: iscsi1 in Connected state
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 service_execute: systemctl mode being used
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 service_execute: Returning exit status [ 0 ]
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 CHECKING IP 127.0.0.1
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 CHECKING IP 172.16.14.199
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 CHECKING IP 172.16.14.47
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 CHECKING IP 10.10.10.1
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 CHECKING IP 10.10.10.3
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:41:47 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:41:47 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:41:47 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:41:47 xen-01 iscsi-ha: 19066 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:41:47 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:41:47 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:41:47 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:41:49 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:50 xen-01 ha-lizard: 17262 Spawning new instance of ha-lizard
Dec 16 09:41:50 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:41:50 xen-01 ha-lizard:  This iteration is count 4639
Dec 16 09:41:50 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:50 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:41:50 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:41:50 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:50 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:50 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:50 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:41:50 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:41:50 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:41:50 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:41:50 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:41:50 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:41:50 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:41:50 xen-01 ha-lizard:  Master management link OK - checking prior link state
Dec 16 09:41:50 xen-01 ha-lizard:  This host detected as pool  Master
Dec 16 09:41:50 xen-01 ha-lizard:  Found 2 hosts in pool
Dec 16 09:41:50 xen-01 ha-lizard:  validate_vm_ha_state: Validating VM HA-state
Dec 16 09:41:50 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] state [ false ] = OK
Dec 16 09:41:50 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] state [ false ] = OK
Dec 16 09:41:50 xen-01 ha-lizard:  Calling function write_pool_state
Dec 16 09:41:50 xen-01 ha-lizard: 19582 Calling function autoselect_slave
Dec 16 09:41:50 xen-01 ha-lizard: 19588 Calling function check_slave_status
Dec 16 09:41:50 xen-01 ha-lizard: 19588 check_master_mgt_link_state: Checking management interface link state
Dec 16 09:41:50 xen-01 ha-lizard: 19582 autoselect_slave: This host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:50 xen-01 ha-lizard: 19582 autoselect_slave: MASTER host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:51 xen-01 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:41:51 xen-01 ha-lizard: 19582 autoselect_slave: 28dfaba4-5b90-4535-afec-a244a6c3f88a is Master UUID - excluding from list of available slaves
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_slave_status: Management link OK - continue
Dec 16 09:41:51 xen-01 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:51 xen-01 ha-lizard: 19582 autoselect_slave: 1 available Slave UUIDs found: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:51 xen-01 ha-lizard: 19588 get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_slave_status: Removing Master UUID from list of Hosts
Dec 16 09:41:51 xen-01 ha-lizard: 19582 autoselect_slave: Selected Slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce = Current slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce - ignoring update
Dec 16 09:41:51 xen-01 ha-lizard: 19588 get_pool_ip_list: returned 172.16.14.200
Dec 16 09:41:51 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:41:51 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:41:51 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_xapi: Pool Host 172.16.14.200 xapi status = 13
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_slave_status: Host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] marked as healthy
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_slave_status: Host IP Address check Status Array for Slaves = (1)
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_slave_status: Quorum check called
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_quorum: Checking host IPs: 172.16.14.199 172.16.14.200
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_quorum: Host IP: 172.16.14.199 Response = OK
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_quorum: LIVE HOSTs = 1
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_quorum: Host IP: 172.16.14.200 Response = OK
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_quorum: LIVE HOSTs = 2
Dec 16 09:41:51 xen-01 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_quorum: Using network points: 172.16.14.165 as possible additional vote
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_quorum: Heuristic IP: 172.16.14.165 Response = OK
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_quorum: Successful Replies = 1
Dec 16 09:41:51 xen-01 ha-lizard: 19588 Total enpoints checked = 1 with total successful replies = 1
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_quorum: Additional heuristic vote success. Incremeting vote by 1
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed
Dec 16 09:41:51 xen-01 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_slave_status: Failed slave count = 0
Dec 16 09:41:51 xen-01 ha-lizard: 19588 check_slave_status: No Failed slaves detected
Dec 16 09:41:51 xen-01 ha-lizard: 19588 Function check_slave_status reported no failures: calling vm_mon
Dec 16 09:41:51 xen-01 ha-lizard: 19588 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Dec 16 09:41:51 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:41:51 xen-01 ha-lizard: 19588 vm_mon: Retrived list of VMs for this poll: 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:51 xen-01 ha-lizard: 19588 vm_mon: Removing Control Domains from VM list
Dec 16 09:41:51 xen-01 ha-lizard: 19588 vm_mon: VM list returned = 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:41:51 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:41:51 xen-01 ha-lizard:  write_status_report: Writing status report
Dec 16 09:41:51 xen-01 ha-lizard: 19588 vm_state: Machine state for 4c4ca253-ff4f-2702-94a2-836fc0380f31 returned: running
Dec 16 09:41:51 xen-01 ha-lizard: 19588 vm_mon: VM 4c4ca253-ff4f-2702-94a2-836fc0380f31 state = running
Dec 16 09:41:51 xen-01 ha-lizard: 19588 vm_state: Machine state for 174112ca-28d4-bbfe-6036-a2ee7a34e444 returned: running
Dec 16 09:41:51 xen-01 ha-lizard: 19588 vm_mon: VM 174112ca-28d4-bbfe-6036-a2ee7a34e444 state = running
Dec 16 09:41:51 xen-01 ha-lizard: 19588 vm_mon: 0 Eligible Halted VMs found
Dec 16 09:41:53 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:41:54 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:41:57 xen-01 iscsi-ha: 19050 Spawning new instance of iscsi-ha
Dec 16 09:41:57 xen-01 iscsi-ha: 19050 check_logger_processes Checking logger processes
Dec 16 09:41:57 xen-01 iscsi-ha: 19050 check_logger_processes No processes to clear
Dec 16 09:41:57 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:41:57 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:41:57 xen-01 iscsi-ha:  This iteration is count 6987
Dec 16 09:41:57 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:41:57 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 service_execute: System V mode detected
Dec 16 09:41:57 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 service_execute: Returning exit status [ 0 ]
Dec 16 09:41:57 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:57 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329436024 nr:3152 dw:4374080 dr:1325355880 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 DRBD Resource: iscsi1 in Connected state
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 service_execute: systemctl mode being used
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 service_execute: Returning exit status [ 0 ]
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 CHECKING IP 127.0.0.1
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 CHECKING IP 172.16.14.199
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 CHECKING IP 172.16.14.47
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 CHECKING IP 10.10.10.1
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 CHECKING IP 10.10.10.3
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:41:57 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:41:57 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:41:57 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:41:57 xen-01 iscsi-ha: 20100 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:41:57 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:41:57 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:41:57 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:41:59 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:03 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:42:04 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:05 xen-01 ha-lizard: 19402 Spawning new instance of ha-lizard
Dec 16 09:42:05 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:42:05 xen-01 ha-lizard:  This iteration is count 4640
Dec 16 09:42:05 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:05 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:42:05 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:42:05 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:05 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:05 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:05 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:42:05 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:42:05 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:42:05 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:42:05 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:42:05 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:42:05 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:42:05 xen-01 ha-lizard:  Master management link OK - checking prior link state
Dec 16 09:42:05 xen-01 ha-lizard:  This host detected as pool  Master
Dec 16 09:42:05 xen-01 ha-lizard:  Found 2 hosts in pool
Dec 16 09:42:05 xen-01 ha-lizard:  validate_vm_ha_state: Validating VM HA-state
Dec 16 09:42:06 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] state [ false ] = OK
Dec 16 09:42:06 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] state [ false ] = OK
Dec 16 09:42:06 xen-01 ha-lizard:  Calling function write_pool_state
Dec 16 09:42:06 xen-01 ha-lizard: 20528 Calling function autoselect_slave
Dec 16 09:42:06 xen-01 ha-lizard: 20534 Calling function check_slave_status
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_master_mgt_link_state: Checking management interface link state
Dec 16 09:42:06 xen-01 ha-lizard: 20528 autoselect_slave: This host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:06 xen-01 ha-lizard: 20528 autoselect_slave: MASTER host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:06 xen-01 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:42:06 xen-01 ha-lizard: 20528 autoselect_slave: 28dfaba4-5b90-4535-afec-a244a6c3f88a is Master UUID - excluding from list of available slaves
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_slave_status: Management link OK - continue
Dec 16 09:42:06 xen-01 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:06 xen-01 ha-lizard: 20534 get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:06 xen-01 ha-lizard: 20528 autoselect_slave: 1 available Slave UUIDs found: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_slave_status: Removing Master UUID from list of Hosts
Dec 16 09:42:06 xen-01 ha-lizard: 20528 autoselect_slave: Selected Slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce = Current slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce - ignoring update
Dec 16 09:42:06 xen-01 ha-lizard: 20534 get_pool_ip_list: returned 172.16.14.200
Dec 16 09:42:06 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:06 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:06 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_xapi: Pool Host 172.16.14.200 xapi status = 13
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_slave_status: Host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] marked as healthy
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_slave_status: Host IP Address check Status Array for Slaves = (1)
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_slave_status: Quorum check called
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_quorum: Checking host IPs: 172.16.14.199 172.16.14.200
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_quorum: Host IP: 172.16.14.199 Response = OK
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_quorum: LIVE HOSTs = 1
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_quorum: Host IP: 172.16.14.200 Response = OK
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_quorum: LIVE HOSTs = 2
Dec 16 09:42:06 xen-01 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_quorum: Using network points: 172.16.14.165 as possible additional vote
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_quorum: Heuristic IP: 172.16.14.165 Response = OK
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_quorum: Successful Replies = 1
Dec 16 09:42:06 xen-01 ha-lizard: 20534 Total enpoints checked = 1 with total successful replies = 1
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_quorum: Additional heuristic vote success. Incremeting vote by 1
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed
Dec 16 09:42:06 xen-01 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_slave_status: Failed slave count = 0
Dec 16 09:42:06 xen-01 ha-lizard: 20534 check_slave_status: No Failed slaves detected
Dec 16 09:42:06 xen-01 ha-lizard: 20534 Function check_slave_status reported no failures: calling vm_mon
Dec 16 09:42:06 xen-01 ha-lizard: 20534 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Dec 16 09:42:06 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:42:06 xen-01 ha-lizard: 20534 vm_mon: Retrived list of VMs for this poll: 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:06 xen-01 ha-lizard: 20534 vm_mon: Removing Control Domains from VM list
Dec 16 09:42:06 xen-01 ha-lizard: 20534 vm_mon: VM list returned = 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:06 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:42:06 xen-01 ha-lizard:  write_status_report: Writing status report
Dec 16 09:42:06 xen-01 ha-lizard: 20534 vm_state: Machine state for 4c4ca253-ff4f-2702-94a2-836fc0380f31 returned: running
Dec 16 09:42:06 xen-01 ha-lizard: 20534 vm_mon: VM 4c4ca253-ff4f-2702-94a2-836fc0380f31 state = running
Dec 16 09:42:06 xen-01 ha-lizard: 20534 vm_state: Machine state for 174112ca-28d4-bbfe-6036-a2ee7a34e444 returned: running
Dec 16 09:42:06 xen-01 ha-lizard: 20534 vm_mon: VM 174112ca-28d4-bbfe-6036-a2ee7a34e444 state = running
Dec 16 09:42:06 xen-01 ha-lizard: 20534 vm_mon: 0 Eligible Halted VMs found
Dec 16 09:42:07 xen-01 iscsi-ha: 20086 Spawning new instance of iscsi-ha
Dec 16 09:42:07 xen-01 iscsi-ha: 20086 check_logger_processes Checking logger processes
Dec 16 09:42:07 xen-01 iscsi-ha: 20086 check_logger_processes No processes to clear
Dec 16 09:42:07 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:42:07 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:42:07 xen-01 iscsi-ha:  This iteration is count 6988
Dec 16 09:42:07 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:07 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 service_execute: System V mode detected
Dec 16 09:42:07 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 service_execute: Returning exit status [ 0 ]
Dec 16 09:42:07 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:07 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329436024 nr:3152 dw:4374080 dr:1325355896 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 DRBD Resource: iscsi1 in Connected state
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 service_execute: systemctl mode being used
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 service_execute: Returning exit status [ 0 ]
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 CHECKING IP 127.0.0.1
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 CHECKING IP 172.16.14.199
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 CHECKING IP 172.16.14.47
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 CHECKING IP 10.10.10.1
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 CHECKING IP 10.10.10.3
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:42:07 xen-01 iscsi-ha: 20995 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:42:07 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:42:07 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:42:07 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:42:08 xen-01 iscsi-ha: 20995 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:42:08 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:42:08 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:42:08 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:42:09 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:13 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:42:14 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:17 xen-01 iscsi-ha: 20981 Spawning new instance of iscsi-ha
Dec 16 09:42:17 xen-01 iscsi-ha: 20981 check_logger_processes Checking logger processes
Dec 16 09:42:17 xen-01 iscsi-ha: 20981 check_logger_processes No processes to clear
Dec 16 09:42:17 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:42:17 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:42:17 xen-01 iscsi-ha:  This iteration is count 6989
Dec 16 09:42:17 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:17 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 service_execute: System V mode detected
Dec 16 09:42:17 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 service_execute: Returning exit status [ 0 ]
Dec 16 09:42:17 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:17 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329436228 nr:3152 dw:4374284 dr:1325355912 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 DRBD Resource: iscsi1 in Connected state
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 service_execute: systemctl mode being used
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 service_execute: Returning exit status [ 0 ]
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:42:17 xen-01 iscsi-ha: 22164 CHECKING IP 127.0.0.1
Dec 16 09:42:18 xen-01 iscsi-ha: 22164 CHECKING IP 172.16.14.199
Dec 16 09:42:18 xen-01 iscsi-ha: 22164 CHECKING IP 172.16.14.47
Dec 16 09:42:18 xen-01 iscsi-ha: 22164 CHECKING IP 10.10.10.1
Dec 16 09:42:18 xen-01 iscsi-ha: 22164 CHECKING IP 10.10.10.3
Dec 16 09:42:18 xen-01 iscsi-ha: 22164 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:42:18 xen-01 iscsi-ha: 22164 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:42:18 xen-01 iscsi-ha: 22164 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:42:18 xen-01 iscsi-ha: 22164 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:42:18 xen-01 iscsi-ha: 22164 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:42:18 xen-01 iscsi-ha: 22164 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:42:18 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:42:18 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:42:18 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:42:18 xen-01 iscsi-ha: 22164 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:42:18 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:42:18 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:42:18 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:42:19 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:20 xen-01 ha-lizard: 20352 Spawning new instance of ha-lizard
Dec 16 09:42:20 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:42:20 xen-01 ha-lizard:  This iteration is count 4641
Dec 16 09:42:20 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:20 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:42:20 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:42:20 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:20 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:20 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:20 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:42:20 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:42:20 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:42:20 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:42:20 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:42:20 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:42:20 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:42:20 xen-01 ha-lizard:  Master management link OK - checking prior link state
Dec 16 09:42:20 xen-01 ha-lizard:  This host detected as pool  Master
Dec 16 09:42:20 xen-01 ha-lizard:  Found 2 hosts in pool
Dec 16 09:42:21 xen-01 ha-lizard:  validate_vm_ha_state: Validating VM HA-state
Dec 16 09:42:21 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] state [ false ] = OK
Dec 16 09:42:21 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] state [ false ] = OK
Dec 16 09:42:21 xen-01 ha-lizard:  Calling function write_pool_state
Dec 16 09:42:21 xen-01 ha-lizard: 22497 Calling function autoselect_slave
Dec 16 09:42:21 xen-01 ha-lizard: 22503 Calling function check_slave_status
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_master_mgt_link_state: Checking management interface link state
Dec 16 09:42:21 xen-01 ha-lizard: 22497 autoselect_slave: This host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:21 xen-01 ha-lizard: 22497 autoselect_slave: MASTER host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:21 xen-01 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:42:21 xen-01 ha-lizard: 22497 autoselect_slave: 28dfaba4-5b90-4535-afec-a244a6c3f88a is Master UUID - excluding from list of available slaves
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_slave_status: Management link OK - continue
Dec 16 09:42:21 xen-01 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:21 xen-01 ha-lizard: 22497 autoselect_slave: 1 available Slave UUIDs found: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:21 xen-01 ha-lizard: 22503 get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_slave_status: Removing Master UUID from list of Hosts
Dec 16 09:42:21 xen-01 ha-lizard: 22497 autoselect_slave: Selected Slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce = Current slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce - ignoring update
Dec 16 09:42:21 xen-01 ha-lizard: 22503 get_pool_ip_list: returned 172.16.14.200
Dec 16 09:42:21 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:21 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:21 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_xapi: Pool Host 172.16.14.200 xapi status = 13
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_slave_status: Host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] marked as healthy
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_slave_status: Host IP Address check Status Array for Slaves = (1)
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_slave_status: Quorum check called
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_quorum: Checking host IPs: 172.16.14.199 172.16.14.200
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_quorum: Host IP: 172.16.14.199 Response = OK
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_quorum: LIVE HOSTs = 1
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_quorum: Host IP: 172.16.14.200 Response = OK
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_quorum: LIVE HOSTs = 2
Dec 16 09:42:21 xen-01 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_quorum: Using network points: 172.16.14.165 as possible additional vote
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_quorum: Heuristic IP: 172.16.14.165 Response = OK
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_quorum: Successful Replies = 1
Dec 16 09:42:21 xen-01 ha-lizard: 22503 Total enpoints checked = 1 with total successful replies = 1
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_quorum: Additional heuristic vote success. Incremeting vote by 1
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed
Dec 16 09:42:21 xen-01 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_slave_status: Failed slave count = 0
Dec 16 09:42:21 xen-01 ha-lizard: 22503 check_slave_status: No Failed slaves detected
Dec 16 09:42:21 xen-01 ha-lizard: 22503 Function check_slave_status reported no failures: calling vm_mon
Dec 16 09:42:21 xen-01 ha-lizard: 22503 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Dec 16 09:42:21 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:42:21 xen-01 ha-lizard: 22503 vm_mon: Retrived list of VMs for this poll: 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:21 xen-01 ha-lizard: 22503 vm_mon: Removing Control Domains from VM list
Dec 16 09:42:21 xen-01 ha-lizard: 22503 vm_mon: VM list returned = 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:21 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:42:21 xen-01 ha-lizard:  write_status_report: Writing status report
Dec 16 09:42:21 xen-01 ha-lizard: 22503 vm_state: Machine state for 4c4ca253-ff4f-2702-94a2-836fc0380f31 returned: running
Dec 16 09:42:21 xen-01 ha-lizard: 22503 vm_mon: VM 4c4ca253-ff4f-2702-94a2-836fc0380f31 state = running
Dec 16 09:42:22 xen-01 ha-lizard: 22503 vm_state: Machine state for 174112ca-28d4-bbfe-6036-a2ee7a34e444 returned: running
Dec 16 09:42:22 xen-01 ha-lizard: 22503 vm_mon: VM 174112ca-28d4-bbfe-6036-a2ee7a34e444 state = running
Dec 16 09:42:22 xen-01 ha-lizard: 22503 vm_mon: 0 Eligible Halted VMs found
Dec 16 09:42:23 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:42:24 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:25 xen-01 perfmon: PERFMON: caught IOError: (http error 401 Unauthorised content-length: 96#015#012content-type:text/html#015#012connection:close#015#012cache-control:no-cache, no-store#015#012WWW-Authenticate:Basic realm="rrd_updates"#015#012) - restarting XAPI session
Dec 16 09:42:27 xen-01 iscsi-ha: 22148 Spawning new instance of iscsi-ha
Dec 16 09:42:27 xen-01 iscsi-ha: 22148 check_logger_processes Checking logger processes
Dec 16 09:42:27 xen-01 iscsi-ha: 22148 check_logger_processes No processes to clear
Dec 16 09:42:27 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:42:27 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:42:27 xen-01 iscsi-ha:  This iteration is count 6990
Dec 16 09:42:27 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:27 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:42:27 xen-01 iscsi-ha: 23198 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:42:27 xen-01 iscsi-ha: 23198 service_execute: System V mode detected
Dec 16 09:42:27 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 service_execute: Returning exit status [ 0 ]
Dec 16 09:42:28 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:28 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329436228 nr:3152 dw:4374284 dr:1325355928 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 DRBD Resource: iscsi1 in Connected state
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 service_execute: systemctl mode being used
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 service_execute: Returning exit status [ 0 ]
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 CHECKING IP 127.0.0.1
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 CHECKING IP 172.16.14.199
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 CHECKING IP 172.16.14.47
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 CHECKING IP 10.10.10.1
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 CHECKING IP 10.10.10.3
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:42:28 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:42:28 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:42:28 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:42:28 xen-01 iscsi-ha: 23198 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:42:28 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:42:28 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:42:28 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:42:29 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:33 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:42:34 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:35 xen-01 ha-lizard: 22321 Spawning new instance of ha-lizard
Dec 16 09:42:35 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:42:35 xen-01 ha-lizard:  This iteration is count 4642
Dec 16 09:42:35 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:35 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:42:35 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:42:35 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:35 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:35 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:35 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:42:35 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:42:35 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:42:35 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:42:36 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:42:36 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:42:36 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:42:36 xen-01 ha-lizard:  Master management link OK - checking prior link state
Dec 16 09:42:36 xen-01 ha-lizard:  This host detected as pool  Master
Dec 16 09:42:36 xen-01 ha-lizard:  Found 2 hosts in pool
Dec 16 09:42:36 xen-01 ha-lizard:  validate_vm_ha_state: Validating VM HA-state
Dec 16 09:42:36 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] state [ false ] = OK
Dec 16 09:42:36 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] state [ false ] = OK
Dec 16 09:42:36 xen-01 ha-lizard:  Calling function write_pool_state
Dec 16 09:42:36 xen-01 ha-lizard: 23620 Calling function autoselect_slave
Dec 16 09:42:36 xen-01 ha-lizard: 23626 Calling function check_slave_status
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_master_mgt_link_state: Checking management interface link state
Dec 16 09:42:36 xen-01 ha-lizard: 23620 autoselect_slave: This host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:36 xen-01 ha-lizard: 23620 autoselect_slave: MASTER host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:36 xen-01 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_slave_status: Management link OK - continue
Dec 16 09:42:36 xen-01 ha-lizard: 23620 autoselect_slave: 28dfaba4-5b90-4535-afec-a244a6c3f88a is Master UUID - excluding from list of available slaves
Dec 16 09:42:36 xen-01 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:36 xen-01 ha-lizard: 23620 autoselect_slave: 1 available Slave UUIDs found: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:36 xen-01 ha-lizard: 23626 get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_slave_status: Removing Master UUID from list of Hosts
Dec 16 09:42:36 xen-01 ha-lizard: 23620 autoselect_slave: Selected Slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce = Current slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce - ignoring update
Dec 16 09:42:36 xen-01 ha-lizard: 23626 get_pool_ip_list: returned 172.16.14.200
Dec 16 09:42:36 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:36 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:36 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_xapi: Pool Host 172.16.14.200 xapi status = 13
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_slave_status: Host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] marked as healthy
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_slave_status: Host IP Address check Status Array for Slaves = (1)
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_slave_status: Quorum check called
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_quorum: Checking host IPs: 172.16.14.199 172.16.14.200
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_quorum: Host IP: 172.16.14.199 Response = OK
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_quorum: LIVE HOSTs = 1
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_quorum: Host IP: 172.16.14.200 Response = OK
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_quorum: LIVE HOSTs = 2
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_quorum: Using network points: 172.16.14.165 as possible additional vote
Dec 16 09:42:36 xen-01 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_quorum: Heuristic IP: 172.16.14.165 Response = OK
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_quorum: Successful Replies = 1
Dec 16 09:42:36 xen-01 ha-lizard: 23626 Total enpoints checked = 1 with total successful replies = 1
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_quorum: Additional heuristic vote success. Incremeting vote by 1
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_slave_status: Failed slave count = 0
Dec 16 09:42:36 xen-01 ha-lizard: 23626 check_slave_status: No Failed slaves detected
Dec 16 09:42:36 xen-01 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:36 xen-01 ha-lizard: 23626 Function check_slave_status reported no failures: calling vm_mon
Dec 16 09:42:36 xen-01 ha-lizard: 23626 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Dec 16 09:42:36 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:42:36 xen-01 ha-lizard: 23626 vm_mon: Retrived list of VMs for this poll: 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:36 xen-01 ha-lizard: 23626 vm_mon: Removing Control Domains from VM list
Dec 16 09:42:36 xen-01 ha-lizard: 23626 vm_mon: VM list returned = 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:36 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:42:36 xen-01 ha-lizard:  write_status_report: Writing status report
Dec 16 09:42:37 xen-01 ha-lizard: 23626 vm_state: Machine state for 4c4ca253-ff4f-2702-94a2-836fc0380f31 returned: running
Dec 16 09:42:37 xen-01 ha-lizard: 23626 vm_mon: VM 4c4ca253-ff4f-2702-94a2-836fc0380f31 state = running
Dec 16 09:42:37 xen-01 ha-lizard: 23626 vm_state: Machine state for 174112ca-28d4-bbfe-6036-a2ee7a34e444 returned: running
Dec 16 09:42:37 xen-01 ha-lizard: 23626 vm_mon: VM 174112ca-28d4-bbfe-6036-a2ee7a34e444 state = running
Dec 16 09:42:37 xen-01 ha-lizard: 23626 vm_mon: 0 Eligible Halted VMs found
Dec 16 09:42:37 xen-01 iscsi-ha: 23184 Spawning new instance of iscsi-ha
Dec 16 09:42:37 xen-01 iscsi-ha: 23184 check_logger_processes Checking logger processes
Dec 16 09:42:38 xen-01 iscsi-ha: 23184 check_logger_processes No processes to clear
Dec 16 09:42:38 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:42:38 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:42:38 xen-01 iscsi-ha:  This iteration is count 6991
Dec 16 09:42:38 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:38 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 service_execute: System V mode detected
Dec 16 09:42:38 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 service_execute: Returning exit status [ 0 ]
Dec 16 09:42:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329436228 nr:3152 dw:4374284 dr:1325355944 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 DRBD Resource: iscsi1 in Connected state
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 service_execute: systemctl mode being used
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 service_execute: Returning exit status [ 0 ]
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 CHECKING IP 127.0.0.1
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 CHECKING IP 172.16.14.199
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 CHECKING IP 172.16.14.47
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 CHECKING IP 10.10.10.1
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 CHECKING IP 10.10.10.3
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:42:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:42:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:42:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:42:38 xen-01 iscsi-ha: 24089 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:42:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:42:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:42:38 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:42:39 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:43 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:42:44 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:48 xen-01 iscsi-ha: 24075 Spawning new instance of iscsi-ha
Dec 16 09:42:48 xen-01 iscsi-ha: 24075 check_logger_processes Checking logger processes
Dec 16 09:42:48 xen-01 iscsi-ha: 24075 check_logger_processes No processes to clear
Dec 16 09:42:48 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:42:48 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:42:48 xen-01 iscsi-ha:  This iteration is count 6992
Dec 16 09:42:48 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:48 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 service_execute: System V mode detected
Dec 16 09:42:48 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 service_execute: Returning exit status [ 0 ]
Dec 16 09:42:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329436368 nr:3152 dw:4374424 dr:1325355960 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 DRBD Resource: iscsi1 in Connected state
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 service_execute: systemctl mode being used
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 service_execute: Returning exit status [ 0 ]
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 CHECKING IP 127.0.0.1
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 CHECKING IP 172.16.14.199
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 CHECKING IP 172.16.14.47
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 CHECKING IP 10.10.10.1
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 CHECKING IP 10.10.10.3
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:42:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:42:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:42:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:42:48 xen-01 iscsi-ha: 24470 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:42:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:42:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:42:48 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:42:49 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:50 xen-01 ha-lizard: 23444 Spawning new instance of ha-lizard
Dec 16 09:42:50 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:42:50 xen-01 ha-lizard:  This iteration is count 4643
Dec 16 09:42:50 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:50 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:42:50 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:42:50 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:50 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:50 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:50 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:42:50 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:42:51 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:42:51 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:42:51 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:42:51 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:42:51 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:42:51 xen-01 ha-lizard:  Master management link OK - checking prior link state
Dec 16 09:42:51 xen-01 ha-lizard:  This host detected as pool  Master
Dec 16 09:42:51 xen-01 ha-lizard:  Found 2 hosts in pool
Dec 16 09:42:51 xen-01 ha-lizard:  validate_vm_ha_state: Validating VM HA-state
Dec 16 09:42:51 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] state [ false ] = OK
Dec 16 09:42:51 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] state [ false ] = OK
Dec 16 09:42:51 xen-01 ha-lizard:  Calling function write_pool_state
Dec 16 09:42:51 xen-01 ha-lizard: 25434 Calling function autoselect_slave
Dec 16 09:42:51 xen-01 ha-lizard: 25439 Calling function check_slave_status
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_master_mgt_link_state: Checking management interface link state
Dec 16 09:42:51 xen-01 ha-lizard: 25434 autoselect_slave: This host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:51 xen-01 ha-lizard: 25434 autoselect_slave: MASTER host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:51 xen-01 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:42:51 xen-01 ha-lizard: 25434 autoselect_slave: 28dfaba4-5b90-4535-afec-a244a6c3f88a is Master UUID - excluding from list of available slaves
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_slave_status: Management link OK - continue
Dec 16 09:42:51 xen-01 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:51 xen-01 ha-lizard: 25434 autoselect_slave: 1 available Slave UUIDs found: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:51 xen-01 ha-lizard: 25439 get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_slave_status: Removing Master UUID from list of Hosts
Dec 16 09:42:51 xen-01 ha-lizard: 25434 autoselect_slave: Selected Slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce = Current slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce - ignoring update
Dec 16 09:42:51 xen-01 ha-lizard: 25439 get_pool_ip_list: returned 172.16.14.200
Dec 16 09:42:51 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:42:51 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:42:51 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_xapi: Pool Host 172.16.14.200 xapi status = 13
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_slave_status: Host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] marked as healthy
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_slave_status: Host IP Address check Status Array for Slaves = (1)
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_slave_status: Quorum check called
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_quorum: Checking host IPs: 172.16.14.199 172.16.14.200
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_quorum: Host IP: 172.16.14.199 Response = OK
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_quorum: LIVE HOSTs = 1
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_quorum: Host IP: 172.16.14.200 Response = OK
Dec 16 09:42:51 xen-01 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_quorum: LIVE HOSTs = 2
Dec 16 09:42:51 xen-01 ha-lizard: 25439 check_quorum: Using network points: 172.16.14.165 as possible additional vote
Dec 16 09:42:52 xen-01 ha-lizard: 25439 check_quorum: Heuristic IP: 172.16.14.165 Response = OK
Dec 16 09:42:52 xen-01 ha-lizard: 25439 check_quorum: Successful Replies = 1
Dec 16 09:42:52 xen-01 ha-lizard: 25439 Total enpoints checked = 1 with total successful replies = 1
Dec 16 09:42:52 xen-01 ha-lizard: 25439 check_quorum: Additional heuristic vote success. Incremeting vote by 1
Dec 16 09:42:52 xen-01 ha-lizard: 25439 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Dec 16 09:42:52 xen-01 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:42:52 xen-01 ha-lizard: 25439 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed
Dec 16 09:42:52 xen-01 ha-lizard: 25439 check_slave_status: Failed slave count = 0
Dec 16 09:42:52 xen-01 ha-lizard: 25439 check_slave_status: No Failed slaves detected
Dec 16 09:42:52 xen-01 ha-lizard: 25439 Function check_slave_status reported no failures: calling vm_mon
Dec 16 09:42:52 xen-01 ha-lizard: 25439 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Dec 16 09:42:52 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:42:52 xen-01 ha-lizard: 25439 vm_mon: Retrived list of VMs for this poll: 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:52 xen-01 ha-lizard: 25439 vm_mon: Removing Control Domains from VM list
Dec 16 09:42:52 xen-01 ha-lizard: 25439 vm_mon: VM list returned = 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:42:52 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:42:52 xen-01 ha-lizard:  write_status_report: Writing status report
Dec 16 09:42:52 xen-01 ha-lizard: 25439 vm_state: Machine state for 4c4ca253-ff4f-2702-94a2-836fc0380f31 returned: running
Dec 16 09:42:52 xen-01 ha-lizard: 25439 vm_mon: VM 4c4ca253-ff4f-2702-94a2-836fc0380f31 state = running
Dec 16 09:42:52 xen-01 ha-lizard: 25439 vm_state: Machine state for 174112ca-28d4-bbfe-6036-a2ee7a34e444 returned: running
Dec 16 09:42:52 xen-01 ha-lizard: 25439 vm_mon: VM 174112ca-28d4-bbfe-6036-a2ee7a34e444 state = running
Dec 16 09:42:52 xen-01 ha-lizard: 25439 vm_mon: 0 Eligible Halted VMs found
Dec 16 09:42:53 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:42:54 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:42:58 xen-01 iscsi-ha: 24436 Spawning new instance of iscsi-ha
Dec 16 09:42:58 xen-01 iscsi-ha: 24436 check_logger_processes Checking logger processes
Dec 16 09:42:58 xen-01 iscsi-ha: 24436 check_logger_processes No processes to clear
Dec 16 09:42:58 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:42:58 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:42:58 xen-01 iscsi-ha:  This iteration is count 6993
Dec 16 09:42:58 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:42:58 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 service_execute: System V mode detected
Dec 16 09:42:58 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 service_execute: Returning exit status [ 0 ]
Dec 16 09:42:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329436368 nr:3152 dw:4374424 dr:1325355976 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 DRBD Resource: iscsi1 in Connected state
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 service_execute: systemctl mode being used
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 service_execute: Returning exit status [ 0 ]
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 CHECKING IP 127.0.0.1
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 CHECKING IP 172.16.14.199
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 CHECKING IP 172.16.14.47
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 CHECKING IP 10.10.10.1
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 CHECKING IP 10.10.10.3
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:42:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:42:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:42:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:42:58 xen-01 iscsi-ha: 26297 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:42:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:42:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:42:58 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:42:59 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:43:03 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:43:04 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:43:05 xen-01 ha-lizard: 25118 Spawning new instance of ha-lizard
Dec 16 09:43:05 xen-01 ha-lizard:  Mail Spool Directory Found /dev/shm/ha-lizard-mail
Dec 16 09:43:05 xen-01 ha-lizard:  This iteration is count 4644
Dec 16 09:43:05 xen-01 ha-lizard:  Checking if this host is a Pool Master or Slave
Dec 16 09:43:05 xen-01 ha-lizard:  This host's pool status = master
Dec 16 09:43:05 xen-01 ha-lizard:  Checking if ha-lizard is enabled for this pool
Dec 16 09:43:06 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:43:06 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:43:06 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:43:06 xen-01 ha-lizard:  ha-lizard is enabled
Dec 16 09:43:06 xen-01 ha-lizard:  check_xs_ha: Checking XenServer HA status
Dec 16 09:43:06 xen-01 ha-lizard:  update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf.
Dec 16 09:43:06 xen-01 ha-lizard:  update_global_conf_params: DISABLED_VAPPS=()#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=172.16.14.165#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=15#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=10#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=10
Dec 16 09:43:06 xen-01 ha-lizard:  check_master_mgt_link_state: Checking management interface link state
Dec 16 09:43:06 xen-01 ha-lizard:  check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:43:06 xen-01 ha-lizard:  check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:43:06 xen-01 ha-lizard:  Master management link OK - checking prior link state
Dec 16 09:43:06 xen-01 ha-lizard:  This host detected as pool  Master
Dec 16 09:43:06 xen-01 ha-lizard:  Found 2 hosts in pool
Dec 16 09:43:06 xen-01 ha-lizard:  validate_vm_ha_state: Validating VM HA-state
Dec 16 09:43:06 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 4c4ca253-ff4f-2702-94a2-836fc0380f31 ] state [ false ] = OK
Dec 16 09:43:06 xen-01 ha-lizard:  validate_vm_ha_state: VM [ 174112ca-28d4-bbfe-6036-a2ee7a34e444 ] state [ false ] = OK
Dec 16 09:43:06 xen-01 ha-lizard:  Calling function write_pool_state
Dec 16 09:43:06 xen-01 ha-lizard: 26707 Calling function autoselect_slave
Dec 16 09:43:06 xen-01 ha-lizard: 26712 Calling function check_slave_status
Dec 16 09:43:06 xen-01 ha-lizard: 26712 check_master_mgt_link_state: Checking management interface link state
Dec 16 09:43:06 xen-01 ha-lizard: 26707 autoselect_slave: This host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:43:06 xen-01 ha-lizard: 26707 autoselect_slave: MASTER host UUID found: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:43:06 xen-01 ha-lizard:  get_vms_on_host: No VMs found on host: 28dfaba4-5b90-4535-afec-a244a6c3f88a
Dec 16 09:43:06 xen-01 ha-lizard: 26712 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 172.16.14.199 ]
Dec 16 09:43:06 xen-01 ha-lizard: 26712 check_master_mgt_link_state: Link [ xenbr4 ] state UP
Dec 16 09:43:06 xen-01 ha-lizard: 26707 autoselect_slave: 28dfaba4-5b90-4535-afec-a244a6c3f88a is Master UUID - excluding from list of available slaves
Dec 16 09:43:06 xen-01 ha-lizard: 26712 check_slave_status: Management link OK - continue
Dec 16 09:43:06 xen-01 ha-lizard:  get_vms_on_host: Returned 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:43:06 xen-01 ha-lizard: 26707 autoselect_slave: 1 available Slave UUIDs found: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:43:06 xen-01 ha-lizard: 26712 get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:43:06 xen-01 ha-lizard: 26712 check_slave_status: Removing Master UUID from list of Hosts
Dec 16 09:43:06 xen-01 ha-lizard: 26707 autoselect_slave: Selected Slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce = Current slave: bcb17f66-0a7a-4aae-bcb4-4f849431a5ce - ignoring update
Dec 16 09:43:06 xen-01 ha-lizard: 26712 get_pool_ip_list: returned 172.16.14.200
Dec 16 09:43:06 xen-01 ha-lizard:  check_ha_enabled: Checking if ha-lizard is enabled for pool: e5038d4e-d066-546d-d5a0-570555096ce6
Dec 16 09:43:06 xen-01 ha-lizard:  check_ha_enabled: ha-lizard is enabled
Dec 16 09:43:06 xen-01 ha-lizard:  check_ha_enabled: checking whether maintenance mode enabled
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_xapi: Pool Host 172.16.14.200 xapi status = 13
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_slave_status: Host [ bcb17f66-0a7a-4aae-bcb4-4f849431a5ce ] marked as healthy
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_slave_status: Host IP Address check Status Array for Slaves = (1)
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_slave_status: Quorum check called
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_quorum: Checking host IPs: 172.16.14.199 172.16.14.200
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_quorum: Host IP: 172.16.14.199 Response = OK
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_quorum: LIVE HOSTs = 1
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_quorum: Host IP: 172.16.14.200 Response = OK
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_quorum: LIVE HOSTs = 2
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_quorum: Using network points: 172.16.14.165 as possible additional vote
Dec 16 09:43:07 xen-01 ha-lizard:  get_pool_host_list: enabled flag set - returning only hosts with enabled=true
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_quorum: Heuristic IP: 172.16.14.165 Response = OK
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_quorum: Successful Replies = 1
Dec 16 09:43:07 xen-01 ha-lizard: 26712 Total enpoints checked = 1 with total successful replies = 1
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_quorum: Additional heuristic vote success. Incremeting vote by 1
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_slave_status: Failed slave count = 0
Dec 16 09:43:07 xen-01 ha-lizard:  get_pool_host_list: returned 28dfaba4-5b90-4535-afec-a244a6c3f88a#012bcb17f66-0a7a-4aae-bcb4-4f849431a5ce
Dec 16 09:43:07 xen-01 ha-lizard: 26712 check_slave_status: No Failed slaves detected
Dec 16 09:43:07 xen-01 ha-lizard: 26712 Function check_slave_status reported no failures: calling vm_mon
Dec 16 09:43:07 xen-01 ha-lizard: 26712 vm_mon: ha-lizard is operating mode 2 - managing pool VMs
Dec 16 09:43:07 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199
Dec 16 09:43:07 xen-01 ha-lizard: 26712 vm_mon: Retrived list of VMs for this poll: 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:43:07 xen-01 ha-lizard: 26712 vm_mon: Removing Control Domains from VM list
Dec 16 09:43:07 xen-01 ha-lizard: 26712 vm_mon: VM list returned = 4c4ca253-ff4f-2702-94a2-836fc0380f31#012174112ca-28d4-bbfe-6036-a2ee7a34e444
Dec 16 09:43:07 xen-01 ha-lizard:  get_pool_ip_list: returned 172.16.14.199 172.16.14.200
Dec 16 09:43:07 xen-01 ha-lizard:  write_status_report: Writing status report
Dec 16 09:43:07 xen-01 ha-lizard: 26712 vm_state: Machine state for 4c4ca253-ff4f-2702-94a2-836fc0380f31 returned: running
Dec 16 09:43:07 xen-01 ha-lizard: 26712 vm_mon: VM 4c4ca253-ff4f-2702-94a2-836fc0380f31 state = running
Dec 16 09:43:07 xen-01 ha-lizard: 26712 vm_state: Machine state for 174112ca-28d4-bbfe-6036-a2ee7a34e444 returned: running
Dec 16 09:43:07 xen-01 ha-lizard: 26712 vm_mon: VM 174112ca-28d4-bbfe-6036-a2ee7a34e444 state = running
Dec 16 09:43:07 xen-01 ha-lizard: 26712 vm_mon: 0 Eligible Halted VMs found
Dec 16 09:43:08 xen-01 iscsi-ha: 26283 Spawning new instance of iscsi-ha
Dec 16 09:43:08 xen-01 iscsi-ha: 26283 check_logger_processes Checking logger processes
Dec 16 09:43:08 xen-01 iscsi-ha: 26283 check_logger_processes No processes to clear
Dec 16 09:43:08 xen-01 iscsi-ha:  XenServer Major Release = [ 7 ]
Dec 16 09:43:08 xen-01 iscsi-ha:  Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Dec 16 09:43:08 xen-01 iscsi-ha:  This iteration is count 6994
Dec 16 09:43:08 xen-01 iscsi-ha:  Checking if this host is a Pool Master or Slave
Dec 16 09:43:08 xen-01 iscsi-ha:  This host's pool status = master
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 service_execute: Execute [ status ] on [ iscsi-ha ]
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 service_execute: System V mode detected
Dec 16 09:43:08 xen-01 iscsi-ha:  auto_plug_pbd: Found LVMoISCSI SR List: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 service_execute: [  OK  ]#015iscsi-ha running: 15590
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 service_execute: Returning exit status [ 0 ]
Dec 16 09:43:08 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:43:08 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: b88adcc4-b907-8315-6aa3-b9d781c78c5e
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 DRBD Running on this host: version: 8.4.3 (api:1/proto:86-101) srcversion: FB3AC7056350AC64629E395 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:1329436368 nr:3152 dw:4374424 dr:1325355992 al:518 bm:211961 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 DRBD Resource: iscsi1 in Connected state
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 service_execute: Execute [ status ] on [ tgtd ]
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 service_execute: systemctl mode being used
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012   Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012  Drop-In: /etc/systemd/system/tgtd.service.d#012           └─local.conf#012   Active: active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago#012 Main PID: 12814 (tgtd)#012   CGroup: /system.slice/tgtd.service#012           └─12814 /usr/sbin/tgtd -f
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 service_execute: Returning exit status [ 0 ]
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 iSCSI target: tgtd status = OK. [ active (running) since Thu 2016-12-15 14:37:13 GMT; 19h ago ]
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 local_ip_list: Local IP list returned 127.0.0.1#012172.16.14.199#012172.16.14.47#01210.10.10.1#01210.10.10.3
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 CHECKING IP 127.0.0.1
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 CHECKING IP 172.16.14.199
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 CHECKING IP 172.16.14.47
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 CHECKING IP 10.10.10.1
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 CHECKING IP 10.10.10.3
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 Virtual IP: 10.10.10.3 discovered on host xen-01
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 send_network_bridge_arp: Processing ARP update for interface [ xenbr6 ]
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 send_network_bridge_arp: This is not a bond - getting device name
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 send_network_bridge_arp: Device name found [ eth6 ]
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 send_network_bridge_arp: IP address list for [ eth6 ] = [ 10.10.10.1#01210.10.10.3 ]
Dec 16 09:43:08 xen-01 iscsi-ha: 27189 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.1 ]
Dec 16 09:43:08 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.1 from 10.10.10.1 eth6
Dec 16 09:43:08 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:43:08 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:43:09 xen-01 iscsi-ha: 27189 send_network_bridge_arp: Updating ARP for device [ eth6 ] IP [ 10.10.10.3 ]
Dec 16 09:43:09 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.10.10.3 from 10.10.10.3 eth6
Dec 16 09:43:09 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 1 probes (1 broadcast(s))
Dec 16 09:43:09 xen-01 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s)
Dec 16 09:43:09 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
Dec 16 09:43:13 xen-01 ha-lizard:  ha-lizard Watchdog: ha-lizard running - OK
Dec 16 09:43:14 xen-01 iscsi-ha:  iscsi-ha Watchdog: iscsi-ha running - OK
