Nov 24 11:32:22 XCP1 iscsi-ha: 23960 conditions not met for replication_link_check Nov 24 11:32:24 XCP1 iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Nov 24 11:32:26 XCP1 ha-lizard: 21935 Spawning new instance of ha-lizard Nov 24 11:32:26 XCP1 ha-lizard: 21935 check_logger_processes: Checking logger processes Nov 24 11:32:26 XCP1 ha-lizard: 21935 check_logger_processes: No processes to clear Nov 24 11:32:26 XCP1 ha-lizard: 24123 LOG_TERMINAL = [false] Nov 24 11:32:26 XCP1 ha-lizard: 24123 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 24 11:32:26 XCP1 ha-lizard: 24123 This iteration is count 3 Nov 24 11:32:26 XCP1 ha-lizard: 24123 Checking if this host is a Pool Master or Slave Nov 24 11:32:26 XCP1 ha-lizard: 24123 This host's pool status = master Nov 24 11:32:26 XCP1 ha-lizard: 24123 Checking if ha-lizard is enabled for this pool Nov 24 11:32:26 XCP1 ha-lizard: 24123 check_ha_enabled: Checking if ha-lizard is enabled for pool: f4849b57-cafb-209a-5ebd-82d89b3e9206 Nov 24 11:32:26 XCP1 ha-lizard: 24123 check_ha_enabled: ha-lizard is enabled Nov 24 11:32:26 XCP1 ha-lizard: 24123 check_ha_enabled: checking whether maintenance mode enabled Nov 24 11:32:27 XCP1 ha-lizard: 24123 ha-lizard is enabled Nov 24 11:32:27 XCP1 ha-lizard: 24123 check_xs_ha: Checking XenServer HA status Nov 24 11:32:27 XCP1 ha-lizard: 24123 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 24 11:32:27 XCP1 ha-lizard: 24123 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=10.1.0.3#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=Devildragon.5#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=20#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=40#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=30 Nov 24 11:32:27 XCP1 ha-lizard: 24123 check_master_mgt_link_state: Checking management interface link state Nov 24 11:32:27 XCP1 ha-lizard: 24123 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 10.1.0.60 ] Nov 24 11:32:27 XCP1 ha-lizard: 24123 check_master_mgt_link_state: Link [ xapi0 ] state UP Nov 24 11:32:27 XCP1 ha-lizard: 24123 Master management link OK - checking prior link state Nov 24 11:32:27 XCP1 ha-lizard: 24123 This host detected as pool Master Nov 24 11:32:27 XCP1 ha-lizard: 24123 Found 2 hosts in pool Nov 24 11:32:27 XCP1 ha-lizard: 24123 validate_vm_ha_state: Validating VM HA-state Nov 24 11:32:27 XCP1 ha-lizard: 24123 Calling function write_pool_state Nov 24 11:32:27 XCP1 ha-lizard: 24123 Calling function autoselect_slave Nov 24 11:32:27 XCP1 ha-lizard: 24123 Calling function check_slave_status Nov 24 11:32:27 XCP1 ha-lizard: 24123 check_master_mgt_link_state: Checking management interface link state Nov 24 11:32:27 XCP1 ha-lizard: 24123 write_pool_state: MASTER UUID found: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:27 XCP1 ha-lizard: 24123 autoselect_slave: This host UUID found: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:27 XCP1 ha-lizard: 24123 write_pool_state: MASTER UUID: 4b1d5376-ca55-4417-bf61-c6afa0360dce written to local state storage Nov 24 11:32:27 XCP1 ha-lizard: 24123 autoselect_slave: MASTER host UUID found: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:27 XCP1 ha-lizard: 24123 write_pool_state: Calling function get_vms_on_host for UUID: 3a600b7e-2711-4403-9e57-6e0f92aecd38 Nov 24 11:32:27 XCP1 ha-lizard: 24123 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 10.1.0.60 ] Nov 24 11:32:27 XCP1 ha-lizard: 24123 check_master_mgt_link_state: Link [ xapi0 ] state UP Nov 24 11:32:27 XCP1 ha-lizard: 24123 check_slave_status: Management link OK - continue Nov 24 11:32:27 XCP1 ha-lizard: 24123 get_vms_on_host: No VMs found on host: 3a600b7e-2711-4403-9e57-6e0f92aecd38 Nov 24 11:32:27 XCP1 ha-lizard: 24123 autoselect_slave: 4b1d5376-ca55-4417-bf61-c6afa0360dce is Master UUID - excluding from list of available slaves Nov 24 11:32:27 XCP1 ha-lizard: 24123 write_pool_state: Writing VM array to local state file host.3a600b7e-2711-4403-9e57-6e0f92aecd38.vmlist.uuid_array Nov 24 11:32:27 XCP1 ha-lizard: 24123 autoselect_slave: 1 available Slave UUIDs found: 3a600b7e-2711-4403-9e57-6e0f92aecd38 Nov 24 11:32:27 XCP1 ha-lizard: 24123 write_pool_state: Calling function get_vms_on_host for UUID: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:27 XCP1 ha-lizard: 24123 get_pool_host_list: returned 3a600b7e-2711-4403-9e57-6e0f92aecd38#0124b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:27 XCP1 ha-lizard: 24123 get_vms_on_host: No VMs found on host: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:27 XCP1 ha-lizard: 24123 write_pool_state: Writing VM array to local state file host.4b1d5376-ca55-4417-bf61-c6afa0360dce.vmlist.uuid_array Nov 24 11:32:27 XCP1 ha-lizard: 24123 autoselect_slave: Selected Slave: 3a600b7e-2711-4403-9e57-6e0f92aecd38 = Current slave: 3a600b7e-2711-4403-9e57-6e0f92aecd38 - ignoring update Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_slave_status: Removing Master UUID from list of Hosts Nov 24 11:32:28 XCP1 ha-lizard: 24123 pool autopromote_uuid = [3a600b7e-2711-4403-9e57-6e0f92aecd38] Nov 24 11:32:28 XCP1 ha-lizard: 24123 get_pool_ip_list: returned 10.1.0.61 Nov 24 11:32:28 XCP1 ha-lizard: 24123 write_pool_state: Pool autopromote_uuid=3a600b7e-2711-4403-9e57-6e0f92aecd38 Nov 24 11:32:28 XCP1 ha-lizard: 24123 write_pool_state: autopromote_uuid unchanged - not updating Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_xapi: Pool Host 10.1.0.61 xapi status = 5 Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_ha_enabled: Checking if ha-lizard is enabled for pool: f4849b57-cafb-209a-5ebd-82d89b3e9206 Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_ha_enabled: ha-lizard is enabled Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_ha_enabled: checking whether maintenance mode enabled Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_slave_status: Host [ 3a600b7e-2711-4403-9e57-6e0f92aecd38 ] marked as healthy Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_slave_status: Host IP Address check Status Array for Slaves = (1) Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_slave_status: Quorum check called Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_quorum: Checking host IPs: 10.1.0.61 10.1.0.60 Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_quorum: Host IP: 10.1.0.61 Response = OK Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_quorum: LIVE HOSTs = 1 Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_quorum: Host IP: 10.1.0.60 Response = OK Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_quorum: LIVE HOSTs = 2 Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_quorum: Using network points: 10.1.0.3 as possible additional vote Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_quorum: Heuristic IP: 10.1.0.3 Response = OK Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_quorum: Successful Replies = 1 Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_quorum: Total enpoints checked = 1 with total successful replies = 1 Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1 Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_slave_status: Failed slave count = 0 Nov 24 11:32:28 XCP1 ha-lizard: 24123 check_slave_status: No Failed slaves detected Nov 24 11:32:28 XCP1 ha-lizard: 24123 write_pool_state: Pool contains 2 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 24 11:32:28 XCP1 ha-lizard: 24123 Function check_slave_status reported no failures: calling vm_mon Nov 24 11:32:28 XCP1 ha-lizard: 24123 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 24 11:32:28 XCP1 ha-lizard: 24123 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Nov 24 11:32:28 XCP1 ha-lizard: 24123 get_pool_host_list: returned 3a600b7e-2711-4403-9e57-6e0f92aecd38#0124b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:28 XCP1 ha-lizard: 24123 vm_mon: Retrived list of VMs for this poll: Nov 24 11:32:28 XCP1 ha-lizard: 24123 get_pool_ip_list: returned 10.1.0.61 Nov 24 11:32:28 XCP1 ha-lizard: 24123 vm_mon: Removing Control Domains from VM list Nov 24 11:32:28 XCP1 ha-lizard: 24123 vm_mon: VM list returned = Nov 24 11:32:28 XCP1 ha-lizard: 24123 vm_mon: 0 Eligible Halted VMs found Nov 24 11:32:28 XCP1 ha-lizard: 24123 get_pool_ip_list: returned 10.1.0.61 10.1.0.60 Nov 24 11:32:28 XCP1 ha-lizard: 24123 write_pool_state: Host IP List = 10.1.0.61 10.1.0.60 Nov 24 11:32:28 XCP1 ha-lizard: 24123 write_status_report: Writing status report Nov 24 11:32:29 XCP1 iscsi-ha: 23944 Spawning new instance of iscsi-ha Nov 24 11:32:29 XCP1 iscsi-ha: 23944 check_logger_processes Checking logger processes Nov 24 11:32:29 XCP1 iscsi-ha: 23944 check_logger_processes No processes to clear Nov 24 11:32:29 XCP1 iscsi-ha: Normalized ISCSI_TARGET_SERVICE [ tgtd ] Nov 24 11:32:29 XCP1 iscsi-ha: XenServer Major Release = [ 8 ] Nov 24 11:32:29 XCP1 iscsi-ha: Mail Spool Directory Found /dev/shm/iscsi-ha-mail Nov 24 11:32:29 XCP1 iscsi-ha: This iteration is count 541 Nov 24 11:32:29 XCP1 iscsi-ha: Checking if this host is a Pool Master or Slave Nov 24 11:32:29 XCP1 iscsi-ha: This host's pool status = master Nov 24 11:32:29 XCP1 iscsi-ha: 24766 service_execute: Execute [ status ] on [ iscsi-ha ] Nov 24 11:32:29 XCP1 iscsi-ha: 24766 service_execute: System V mode detected Nov 24 11:32:29 XCP1 iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Nov 24 11:32:29 XCP1 iscsi-ha: auto_plug_pbd: Found LVMoISCSI SR List: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:32:29 XCP1 iscsi-ha: 24766 service_execute: [ OK ]#015iscsi-ha running: 3968 Nov 24 11:32:29 XCP1 iscsi-ha: 24766 service_execute: Returning exit status [ 0 ] Nov 24 11:32:29 XCP1 iscsi-ha: 24766 DRBD Running on this host: version: 8.4.10 (api:1/proto:86-101) srcversion: CE41F2FAB70AE3534BF40A6 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- ns:157848968 nr:0 dw:10372 dr:157841992 al:2 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:172799500 [========>...........] sync'ed: 47.8% (168748/322888)M finish: 1:40:12 speed: 28,728 (28,848) K/sec Nov 24 11:32:29 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:32:29 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:32:29 XCP1 iscsi-ha: 24766 validate_drbd_resources_loaded: Checking DRBD has loaded with resources. Checking [ 7 ] > [ 2 ] Nov 24 11:32:29 XCP1 iscsi-ha: 24766 validate_drbd_resources_loaded: Resources loaded Nov 24 11:32:29 XCP1 iscsi-ha: 24766 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode Nov 24 11:32:29 XCP1 iscsi-ha: 24766 DRBD resource: iscsi1 is currently syncing [SyncSource] Nov 24 11:32:29 XCP1 iscsi-ha: 24766 service_execute: Execute [ status ] on [ tgtd ] Nov 24 11:32:29 XCP1 iscsi-ha: 24766 service_execute: systemctl mode being used Nov 24 11:32:29 XCP1 iscsi-ha: 24766 service_execute: ? tgtd.service - tgtd iSCSI target daemon#012 Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012 Drop-In: /etc/systemd/system/tgtd.service.d#012 L¦local.conf#012 Active: active (running) since Tue 2020-11-24 10:01:23 CET; 1h 31min ago#012 Process: 4859 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)#012 Process: 4827 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)#012 Process: 4825 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)#012 Process: 4734 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS)#012 Main PID: 4733 (tgtd)#012 CGroup: /system.slice/tgtd.service#012 L¦4733 /usr/sbin/tgtd -f Nov 24 11:32:29 XCP1 iscsi-ha: 24766 service_execute: Returning exit status [ 0 ] Nov 24 11:32:29 XCP1 iscsi-ha: 24766 iSCSI target: tgtd status = OK. [ active (running) since Tue 2020-11-24 10:01:23 CET; 1h 31min ago ] Nov 24 11:32:29 XCP1 iscsi-ha: 24766 local_ip_list: Local IP list returned 127.0.0.1#01210.1.0.60#01210.1.10.2#01210.1.10.4 Nov 24 11:32:29 XCP1 iscsi-ha: 24766 CHECKING IP 127.0.0.1 Nov 24 11:32:29 XCP1 iscsi-ha: 24766 CHECKING IP 10.1.0.60 Nov 24 11:32:29 XCP1 iscsi-ha: 24766 CHECKING IP 10.1.10.2 Nov 24 11:32:29 XCP1 iscsi-ha: 24766 CHECKING IP 10.1.10.4 Nov 24 11:32:30 XCP1 iscsi-ha: 24766 Virtual IP: 10.1.10.4 discovered on host XCP1 Nov 24 11:32:30 XCP1 iscsi-ha: 24766 send_replication_network_arp: Sending ARP update to peer Nov 24 11:32:30 XCP1 iscsi-ha: 24766 send_replication_network_arp: IP address list for [ xapi1 ] = [ 10.1.10.2#01210.1.10.4 ] Nov 24 11:32:30 XCP1 iscsi-ha: 24766 send_replication_network_arp: Updating ARP for device [ xapi1 ] IP [ 10.1.10.2 ] Nov 24 11:32:31 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.1.10.2 from 10.1.10.2 xapi1 Nov 24 11:32:31 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 2 probes (2 broadcast(s)) Nov 24 11:32:31 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s) Nov 24 11:32:31 XCP1 iscsi-ha: 24766 send_replication_network_arp: Updating ARP for device [ xapi1 ] IP [ 10.1.10.4 ] Nov 24 11:32:31 XCP1 ha-lizard: 1843 ha-lizard Watchdog: ha-lizard running - OK Nov 24 11:32:32 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.1.10.4 from 10.1.10.4 xapi1 Nov 24 11:32:32 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 2 probes (2 broadcast(s)) Nov 24 11:32:32 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s) Nov 24 11:32:32 XCP1 iscsi-ha: 24766 Skipping replication_link_check DRBD_LINK_OK=[true] STATE=[master] MANUAL_MODE=[off] Nov 24 11:32:34 XCP1 iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Nov 24 11:32:39 XCP1 iscsi-ha: 24749 Spawning new instance of iscsi-ha Nov 24 11:32:39 XCP1 iscsi-ha: 24749 check_logger_processes Checking logger processes Nov 24 11:32:39 XCP1 iscsi-ha: 24749 check_logger_processes No processes to clear Nov 24 11:32:39 XCP1 iscsi-ha: Normalized ISCSI_TARGET_SERVICE [ tgtd ] Nov 24 11:32:39 XCP1 iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Nov 24 11:32:39 XCP1 iscsi-ha: XenServer Major Release = [ 8 ] Nov 24 11:32:39 XCP1 iscsi-ha: Mail Spool Directory Found /dev/shm/iscsi-ha-mail Nov 24 11:32:39 XCP1 iscsi-ha: This iteration is count 542 Nov 24 11:32:39 XCP1 iscsi-ha: Checking if this host is a Pool Master or Slave Nov 24 11:32:39 XCP1 iscsi-ha: This host's pool status = master Nov 24 11:32:39 XCP1 iscsi-ha: 25034 service_execute: Execute [ status ] on [ iscsi-ha ] Nov 24 11:32:39 XCP1 iscsi-ha: 25034 service_execute: System V mode detected Nov 24 11:32:39 XCP1 iscsi-ha: auto_plug_pbd: Found LVMoISCSI SR List: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:32:39 XCP1 iscsi-ha: 25034 service_execute: [ OK ]#015iscsi-ha running: 3968 Nov 24 11:32:39 XCP1 iscsi-ha: 25034 service_execute: Returning exit status [ 0 ] Nov 24 11:32:39 XCP1 iscsi-ha: 25034 DRBD Running on this host: version: 8.4.10 (api:1/proto:86-101) srcversion: CE41F2FAB70AE3534BF40A6 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- ns:158142168 nr:0 dw:10372 dr:158135620 al:2 bm:0 lo:2 pe:7 ua:3 ap:0 ep:1 wo:f oos:172508616 [========>...........] sync'ed: 47.9% (168464/322888)M finish: 1:38:32 speed: 29,172 (28,848) K/sec Nov 24 11:32:39 XCP1 iscsi-ha: 25034 validate_drbd_resources_loaded: Checking DRBD has loaded with resources. Checking [ 7 ] > [ 2 ] Nov 24 11:32:39 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:32:39 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:32:39 XCP1 iscsi-ha: 25034 validate_drbd_resources_loaded: Resources loaded Nov 24 11:32:39 XCP1 iscsi-ha: 25034 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode Nov 24 11:32:39 XCP1 iscsi-ha: 25034 DRBD resource: iscsi1 is currently syncing [SyncSource] Nov 24 11:32:39 XCP1 iscsi-ha: 25034 service_execute: Execute [ status ] on [ tgtd ] Nov 24 11:32:40 XCP1 iscsi-ha: 25034 service_execute: systemctl mode being used Nov 24 11:32:40 XCP1 iscsi-ha: 25034 service_execute: ? tgtd.service - tgtd iSCSI target daemon#012 Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012 Drop-In: /etc/systemd/system/tgtd.service.d#012 L¦local.conf#012 Active: active (running) since Tue 2020-11-24 10:01:23 CET; 1h 31min ago#012 Process: 4859 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)#012 Process: 4827 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)#012 Process: 4825 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)#012 Process: 4734 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS)#012 Main PID: 4733 (tgtd)#012 CGroup: /system.slice/tgtd.service#012 L¦4733 /usr/sbin/tgtd -f Nov 24 11:32:40 XCP1 iscsi-ha: 25034 service_execute: Returning exit status [ 0 ] Nov 24 11:32:40 XCP1 iscsi-ha: 25034 iSCSI target: tgtd status = OK. [ active (running) since Tue 2020-11-24 10:01:23 CET; 1h 31min ago ] Nov 24 11:32:40 XCP1 iscsi-ha: 25034 local_ip_list: Local IP list returned 127.0.0.1#01210.1.0.60#01210.1.10.2#01210.1.10.4 Nov 24 11:32:40 XCP1 iscsi-ha: 25034 CHECKING IP 127.0.0.1 Nov 24 11:32:40 XCP1 iscsi-ha: 25034 CHECKING IP 10.1.0.60 Nov 24 11:32:40 XCP1 iscsi-ha: 25034 CHECKING IP 10.1.10.2 Nov 24 11:32:40 XCP1 iscsi-ha: 25034 CHECKING IP 10.1.10.4 Nov 24 11:32:40 XCP1 iscsi-ha: 25034 Virtual IP: 10.1.10.4 discovered on host XCP1 Nov 24 11:32:40 XCP1 iscsi-ha: 25034 send_replication_network_arp: Sending ARP update to peer Nov 24 11:32:40 XCP1 iscsi-ha: 25034 send_replication_network_arp: IP address list for [ xapi1 ] = [ 10.1.10.2#01210.1.10.4 ] Nov 24 11:32:40 XCP1 iscsi-ha: 25034 send_replication_network_arp: Updating ARP for device [ xapi1 ] IP [ 10.1.10.2 ] Nov 24 11:32:41 XCP1 ha-lizard: 1843 ha-lizard Watchdog: ha-lizard running - OK Nov 24 11:32:41 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.1.10.2 from 10.1.10.2 xapi1 Nov 24 11:32:41 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 2 probes (2 broadcast(s)) Nov 24 11:32:41 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s) Nov 24 11:32:41 XCP1 iscsi-ha: 25034 send_replication_network_arp: Updating ARP for device [ xapi1 ] IP [ 10.1.10.4 ] Nov 24 11:32:42 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.1.10.4 from 10.1.10.4 xapi1 Nov 24 11:32:42 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 2 probes (2 broadcast(s)) Nov 24 11:32:42 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s) Nov 24 11:32:42 XCP1 iscsi-ha: 25034 Skipping replication_link_check DRBD_LINK_OK=[true] STATE=[master] MANUAL_MODE=[off] Nov 24 11:32:44 XCP1 iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Nov 24 11:32:46 XCP1 ha-lizard: 21935 Spawning new instance of ha-lizard Nov 24 11:32:46 XCP1 ha-lizard: 21935 check_logger_processes: Checking logger processes Nov 24 11:32:46 XCP1 ha-lizard: 21935 check_logger_processes: No processes to clear Nov 24 11:32:46 XCP1 ha-lizard: 25195 LOG_TERMINAL = [false] Nov 24 11:32:46 XCP1 ha-lizard: 25195 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 24 11:32:46 XCP1 ha-lizard: 25195 This iteration is count 4 Nov 24 11:32:47 XCP1 ha-lizard: 25195 Checking if this host is a Pool Master or Slave Nov 24 11:32:47 XCP1 ha-lizard: 25195 This host's pool status = master Nov 24 11:32:47 XCP1 ha-lizard: 25195 Checking if ha-lizard is enabled for this pool Nov 24 11:32:47 XCP1 ha-lizard: 25195 check_ha_enabled: Checking if ha-lizard is enabled for pool: f4849b57-cafb-209a-5ebd-82d89b3e9206 Nov 24 11:32:47 XCP1 ha-lizard: 25195 check_ha_enabled: ha-lizard is enabled Nov 24 11:32:47 XCP1 ha-lizard: 25195 check_ha_enabled: checking whether maintenance mode enabled Nov 24 11:32:47 XCP1 ha-lizard: 25195 ha-lizard is enabled Nov 24 11:32:47 XCP1 ha-lizard: 25195 check_xs_ha: Checking XenServer HA status Nov 24 11:32:47 XCP1 ha-lizard: 25195 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 24 11:32:47 XCP1 ha-lizard: 25195 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=10.1.0.3#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=Devildragon.5#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=20#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=40#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=30 Nov 24 11:32:47 XCP1 ha-lizard: 25195 check_master_mgt_link_state: Checking management interface link state Nov 24 11:32:47 XCP1 ha-lizard: 25195 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 10.1.0.60 ] Nov 24 11:32:47 XCP1 ha-lizard: 25195 check_master_mgt_link_state: Link [ xapi0 ] state UP Nov 24 11:32:47 XCP1 ha-lizard: 25195 Master management link OK - checking prior link state Nov 24 11:32:47 XCP1 ha-lizard: 25195 This host detected as pool Master Nov 24 11:32:47 XCP1 ha-lizard: 25195 Found 2 hosts in pool Nov 24 11:32:47 XCP1 ha-lizard: 25195 validate_vm_ha_state: Validating VM HA-state Nov 24 11:32:47 XCP1 ha-lizard: 25195 Calling function write_pool_state Nov 24 11:32:47 XCP1 ha-lizard: 25195 Calling function autoselect_slave Nov 24 11:32:47 XCP1 ha-lizard: 25195 Calling function check_slave_status Nov 24 11:32:47 XCP1 ha-lizard: 25195 check_master_mgt_link_state: Checking management interface link state Nov 24 11:32:47 XCP1 ha-lizard: 25195 write_pool_state: MASTER UUID found: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:47 XCP1 ha-lizard: 25195 autoselect_slave: This host UUID found: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:47 XCP1 ha-lizard: 25195 write_pool_state: MASTER UUID: 4b1d5376-ca55-4417-bf61-c6afa0360dce written to local state storage Nov 24 11:32:47 XCP1 ha-lizard: 25195 autoselect_slave: MASTER host UUID found: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:47 XCP1 ha-lizard: 25195 write_pool_state: Calling function get_vms_on_host for UUID: 3a600b7e-2711-4403-9e57-6e0f92aecd38 Nov 24 11:32:47 XCP1 ha-lizard: 25195 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 10.1.0.60 ] Nov 24 11:32:47 XCP1 ha-lizard: 25195 check_master_mgt_link_state: Link [ xapi0 ] state UP Nov 24 11:32:47 XCP1 ha-lizard: 25195 check_slave_status: Management link OK - continue Nov 24 11:32:48 XCP1 ha-lizard: 25195 get_vms_on_host: No VMs found on host: 3a600b7e-2711-4403-9e57-6e0f92aecd38 Nov 24 11:32:48 XCP1 ha-lizard: 25195 autoselect_slave: 4b1d5376-ca55-4417-bf61-c6afa0360dce is Master UUID - excluding from list of available slaves Nov 24 11:32:48 XCP1 ha-lizard: 25195 write_pool_state: Writing VM array to local state file host.3a600b7e-2711-4403-9e57-6e0f92aecd38.vmlist.uuid_array Nov 24 11:32:48 XCP1 ha-lizard: 25195 autoselect_slave: 1 available Slave UUIDs found: 3a600b7e-2711-4403-9e57-6e0f92aecd38 Nov 24 11:32:48 XCP1 ha-lizard: 25195 get_pool_host_list: returned 3a600b7e-2711-4403-9e57-6e0f92aecd38#0124b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:48 XCP1 ha-lizard: 25195 write_pool_state: Calling function get_vms_on_host for UUID: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:48 XCP1 ha-lizard: 25195 get_vms_on_host: No VMs found on host: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:48 XCP1 ha-lizard: 25195 autoselect_slave: Selected Slave: 3a600b7e-2711-4403-9e57-6e0f92aecd38 = Current slave: 3a600b7e-2711-4403-9e57-6e0f92aecd38 - ignoring update Nov 24 11:32:48 XCP1 ha-lizard: 25195 write_pool_state: Writing VM array to local state file host.4b1d5376-ca55-4417-bf61-c6afa0360dce.vmlist.uuid_array Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_slave_status: Removing Master UUID from list of Hosts Nov 24 11:32:48 XCP1 ha-lizard: 25195 get_pool_ip_list: returned 10.1.0.61 Nov 24 11:32:48 XCP1 ha-lizard: 25195 pool autopromote_uuid = [3a600b7e-2711-4403-9e57-6e0f92aecd38] Nov 24 11:32:48 XCP1 ha-lizard: 25195 write_pool_state: Pool autopromote_uuid=3a600b7e-2711-4403-9e57-6e0f92aecd38 Nov 24 11:32:48 XCP1 ha-lizard: 25195 write_pool_state: autopromote_uuid unchanged - not updating Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_xapi: Pool Host 10.1.0.61 xapi status = 5 Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_ha_enabled: Checking if ha-lizard is enabled for pool: f4849b57-cafb-209a-5ebd-82d89b3e9206 Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_slave_status: Host [ 3a600b7e-2711-4403-9e57-6e0f92aecd38 ] marked as healthy Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_ha_enabled: ha-lizard is enabled Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_slave_status: Host IP Address check Status Array for Slaves = (1) Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_ha_enabled: checking whether maintenance mode enabled Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_slave_status: Quorum check called Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_quorum: Checking host IPs: 10.1.0.61 10.1.0.60 Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_quorum: Host IP: 10.1.0.61 Response = OK Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_quorum: LIVE HOSTs = 1 Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_quorum: Host IP: 10.1.0.60 Response = OK Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_quorum: LIVE HOSTs = 2 Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_quorum: Using network points: 10.1.0.3 as possible additional vote Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_quorum: Heuristic IP: 10.1.0.3 Response = OK Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_quorum: Successful Replies = 1 Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_quorum: Total enpoints checked = 1 with total successful replies = 1 Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1 Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_slave_status: Failed slave count = 0 Nov 24 11:32:48 XCP1 ha-lizard: 25195 check_slave_status: No Failed slaves detected Nov 24 11:32:48 XCP1 ha-lizard: 25195 Function check_slave_status reported no failures: calling vm_mon Nov 24 11:32:48 XCP1 ha-lizard: 25195 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Nov 24 11:32:48 XCP1 ha-lizard: 25195 write_pool_state: Pool contains 2 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 24 11:32:48 XCP1 ha-lizard: 25195 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 24 11:32:48 XCP1 ha-lizard: 25195 vm_mon: Retrived list of VMs for this poll: Nov 24 11:32:48 XCP1 ha-lizard: 25195 vm_mon: Removing Control Domains from VM list Nov 24 11:32:48 XCP1 ha-lizard: 25195 get_pool_host_list: returned 3a600b7e-2711-4403-9e57-6e0f92aecd38#0124b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:32:48 XCP1 ha-lizard: 25195 vm_mon: VM list returned = Nov 24 11:32:48 XCP1 ha-lizard: 25195 vm_mon: 0 Eligible Halted VMs found Nov 24 11:32:48 XCP1 ha-lizard: 25195 get_pool_ip_list: returned 10.1.0.61 Nov 24 11:32:48 XCP1 ha-lizard: 25195 get_pool_ip_list: returned 10.1.0.61 10.1.0.60 Nov 24 11:32:48 XCP1 ha-lizard: 25195 write_pool_state: Host IP List = 10.1.0.61 10.1.0.60 Nov 24 11:32:48 XCP1 ha-lizard: 25195 write_status_report: Writing status report Nov 24 11:32:49 XCP1 iscsi-ha: 25014 Spawning new instance of iscsi-ha Nov 24 11:32:49 XCP1 iscsi-ha: 25014 check_logger_processes Checking logger processes Nov 24 11:32:49 XCP1 iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Nov 24 11:32:49 XCP1 iscsi-ha: 25014 check_logger_processes No processes to clear Nov 24 11:32:49 XCP1 iscsi-ha: Normalized ISCSI_TARGET_SERVICE [ tgtd ] Nov 24 11:32:49 XCP1 iscsi-ha: XenServer Major Release = [ 8 ] Nov 24 11:32:49 XCP1 iscsi-ha: Mail Spool Directory Found /dev/shm/iscsi-ha-mail Nov 24 11:32:49 XCP1 iscsi-ha: This iteration is count 543 Nov 24 11:32:49 XCP1 iscsi-ha: Checking if this host is a Pool Master or Slave Nov 24 11:32:49 XCP1 iscsi-ha: This host's pool status = master Nov 24 11:32:49 XCP1 iscsi-ha: 25845 service_execute: Execute [ status ] on [ iscsi-ha ] Nov 24 11:32:49 XCP1 iscsi-ha: 25845 service_execute: System V mode detected Nov 24 11:32:49 XCP1 iscsi-ha: auto_plug_pbd: Found LVMoISCSI SR List: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:32:50 XCP1 iscsi-ha: 25845 service_execute: [ OK ]#015iscsi-ha running: 3968 Nov 24 11:32:50 XCP1 iscsi-ha: 25845 service_execute: Returning exit status [ 0 ] Nov 24 11:32:50 XCP1 iscsi-ha: 25845 DRBD Running on this host: version: 8.4.10 (api:1/proto:86-101) srcversion: CE41F2FAB70AE3534BF40A6 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- ns:158427004 nr:0 dw:10372 dr:158420028 al:2 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:172221464 [========>...........] sync'ed: 48.0% (168184/322888)M finish: 1:37:02 speed: 29,556 (28,848) K/sec Nov 24 11:32:50 XCP1 iscsi-ha: 25845 validate_drbd_resources_loaded: Checking DRBD has loaded with resources. Checking [ 7 ] > [ 2 ] Nov 24 11:32:50 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:32:50 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:32:50 XCP1 iscsi-ha: 25845 validate_drbd_resources_loaded: Resources loaded Nov 24 11:32:50 XCP1 iscsi-ha: 25845 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode Nov 24 11:32:50 XCP1 iscsi-ha: 25845 DRBD resource: iscsi1 is currently syncing [SyncSource] Nov 24 11:32:50 XCP1 iscsi-ha: 25845 service_execute: Execute [ status ] on [ tgtd ] Nov 24 11:32:50 XCP1 iscsi-ha: 25845 service_execute: systemctl mode being used Nov 24 11:32:50 XCP1 iscsi-ha: 25845 service_execute: ? tgtd.service - tgtd iSCSI target daemon#012 Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012 Drop-In: /etc/systemd/system/tgtd.service.d#012 L¦local.conf#012 Active: active (running) since Tue 2020-11-24 10:01:23 CET; 1h 31min ago#012 Process: 4859 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)#012 Process: 4827 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)#012 Process: 4825 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)#012 Process: 4734 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS)#012 Main PID: 4733 (tgtd)#012 CGroup: /system.slice/tgtd.service#012 L¦4733 /usr/sbin/tgtd -f Nov 24 11:32:50 XCP1 iscsi-ha: 25845 service_execute: Returning exit status [ 0 ] Nov 24 11:32:50 XCP1 iscsi-ha: 25845 iSCSI target: tgtd status = OK. [ active (running) since Tue 2020-11-24 10:01:23 CET; 1h 31min ago ] Nov 24 11:32:50 XCP1 iscsi-ha: 25845 local_ip_list: Local IP list returned 127.0.0.1#01210.1.0.60#01210.1.10.2#01210.1.10.4 Nov 24 11:32:50 XCP1 iscsi-ha: 25845 CHECKING IP 127.0.0.1 Nov 24 11:32:50 XCP1 iscsi-ha: 25845 CHECKING IP 10.1.0.60 Nov 24 11:32:50 XCP1 iscsi-ha: 25845 CHECKING IP 10.1.10.2 Nov 24 11:32:50 XCP1 iscsi-ha: 25845 CHECKING IP 10.1.10.4 Nov 24 11:32:50 XCP1 iscsi-ha: 25845 Virtual IP: 10.1.10.4 discovered on host XCP1 Nov 24 11:32:50 XCP1 iscsi-ha: 25845 send_replication_network_arp: Sending ARP update to peer Nov 24 11:32:50 XCP1 iscsi-ha: 25845 send_replication_network_arp: IP address list for [ xapi1 ] = [ 10.1.10.2#01210.1.10.4 ] Nov 24 11:32:50 XCP1 iscsi-ha: 25845 send_replication_network_arp: Updating ARP for device [ xapi1 ] IP [ 10.1.10.2 ] Nov 24 11:32:51 XCP1 ha-lizard: 1843 ha-lizard Watchdog: ha-lizard running - OK Nov 24 11:32:51 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.1.10.2 from 10.1.10.2 xapi1 Nov 24 11:32:51 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 2 probes (2 broadcast(s)) Nov 24 11:32:51 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s) Nov 24 11:32:51 XCP1 iscsi-ha: 25845 send_replication_network_arp: Updating ARP for device [ xapi1 ] IP [ 10.1.10.4 ] Nov 24 11:32:52 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.1.10.4 from 10.1.10.4 xapi1 Nov 24 11:32:52 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 2 probes (2 broadcast(s)) Nov 24 11:32:52 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s) Nov 24 11:32:52 XCP1 iscsi-ha: 25845 Skipping replication_link_check DRBD_LINK_OK=[true] STATE=[master] MANUAL_MODE=[off] Nov 24 11:32:54 XCP1 iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Nov 24 11:32:59 XCP1 iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Nov 24 11:32:59 XCP1 iscsi-ha: 25827 Spawning new instance of iscsi-ha Nov 24 11:32:59 XCP1 iscsi-ha: 25827 check_logger_processes Checking logger processes Nov 24 11:32:59 XCP1 iscsi-ha: 25827 check_logger_processes No processes to clear Nov 24 11:32:59 XCP1 iscsi-ha: Normalized ISCSI_TARGET_SERVICE [ tgtd ] Nov 24 11:32:59 XCP1 iscsi-ha: XenServer Major Release = [ 8 ] Nov 24 11:32:59 XCP1 iscsi-ha: Mail Spool Directory Found /dev/shm/iscsi-ha-mail Nov 24 11:32:59 XCP1 iscsi-ha: This iteration is count 544 Nov 24 11:32:59 XCP1 iscsi-ha: Checking if this host is a Pool Master or Slave Nov 24 11:32:59 XCP1 iscsi-ha: This host's pool status = master Nov 24 11:32:59 XCP1 iscsi-ha: 26106 service_execute: Execute [ status ] on [ iscsi-ha ] Nov 24 11:32:59 XCP1 iscsi-ha: 26106 service_execute: System V mode detected Nov 24 11:33:00 XCP1 iscsi-ha: auto_plug_pbd: Found LVMoISCSI SR List: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:33:00 XCP1 iscsi-ha: 26106 service_execute: [ OK ]#015iscsi-ha running: 3968 Nov 24 11:33:00 XCP1 iscsi-ha: 26106 service_execute: Returning exit status [ 0 ] Nov 24 11:33:00 XCP1 iscsi-ha: 26106 DRBD Running on this host: version: 8.4.10 (api:1/proto:86-101) srcversion: CE41F2FAB70AE3534BF40A6 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- ns:158701936 nr:0 dw:10372 dr:158694960 al:2 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:171946532 [========>...........] sync'ed: 48.0% (167916/322888)M finish: 1:39:46 speed: 28,716 (28,844) K/sec Nov 24 11:33:00 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:33:00 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:33:00 XCP1 iscsi-ha: 26106 validate_drbd_resources_loaded: Checking DRBD has loaded with resources. Checking [ 7 ] > [ 2 ] Nov 24 11:33:00 XCP1 iscsi-ha: 26106 validate_drbd_resources_loaded: Resources loaded Nov 24 11:33:00 XCP1 iscsi-ha: 26106 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode Nov 24 11:33:00 XCP1 iscsi-ha: 26106 DRBD resource: iscsi1 is currently syncing [SyncSource] Nov 24 11:33:00 XCP1 iscsi-ha: 26106 service_execute: Execute [ status ] on [ tgtd ] Nov 24 11:33:00 XCP1 iscsi-ha: 26106 service_execute: systemctl mode being used Nov 24 11:33:00 XCP1 iscsi-ha: 26106 service_execute: ? tgtd.service - tgtd iSCSI target daemon#012 Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012 Drop-In: /etc/systemd/system/tgtd.service.d#012 L¦local.conf#012 Active: active (running) since Tue 2020-11-24 10:01:23 CET; 1h 31min ago#012 Process: 4859 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)#012 Process: 4827 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)#012 Process: 4825 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)#012 Process: 4734 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS)#012 Main PID: 4733 (tgtd)#012 CGroup: /system.slice/tgtd.service#012 L¦4733 /usr/sbin/tgtd -f Nov 24 11:33:00 XCP1 iscsi-ha: 26106 service_execute: Returning exit status [ 0 ] Nov 24 11:33:00 XCP1 iscsi-ha: 26106 iSCSI target: tgtd status = OK. [ active (running) since Tue 2020-11-24 10:01:23 CET; 1h 31min ago ] Nov 24 11:33:00 XCP1 iscsi-ha: 26106 local_ip_list: Local IP list returned 127.0.0.1#01210.1.0.60#01210.1.10.2#01210.1.10.4 Nov 24 11:33:00 XCP1 iscsi-ha: 26106 CHECKING IP 127.0.0.1 Nov 24 11:33:00 XCP1 iscsi-ha: 26106 CHECKING IP 10.1.0.60 Nov 24 11:33:00 XCP1 iscsi-ha: 26106 CHECKING IP 10.1.10.2 Nov 24 11:33:00 XCP1 iscsi-ha: 26106 CHECKING IP 10.1.10.4 Nov 24 11:33:00 XCP1 iscsi-ha: 26106 Virtual IP: 10.1.10.4 discovered on host XCP1 Nov 24 11:33:00 XCP1 iscsi-ha: 26106 send_replication_network_arp: Sending ARP update to peer Nov 24 11:33:00 XCP1 iscsi-ha: 26106 send_replication_network_arp: IP address list for [ xapi1 ] = [ 10.1.10.2#01210.1.10.4 ] Nov 24 11:33:00 XCP1 iscsi-ha: 26106 send_replication_network_arp: Updating ARP for device [ xapi1 ] IP [ 10.1.10.2 ] Nov 24 11:33:01 XCP1 ha-lizard: 1843 ha-lizard Watchdog: ha-lizard running - OK Nov 24 11:33:01 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.1.10.2 from 10.1.10.2 xapi1 Nov 24 11:33:01 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 2 probes (2 broadcast(s)) Nov 24 11:33:01 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s) Nov 24 11:33:01 XCP1 iscsi-ha: 26106 send_replication_network_arp: Updating ARP for device [ xapi1 ] IP [ 10.1.10.4 ] Nov 24 11:33:02 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.1.10.4 from 10.1.10.4 xapi1 Nov 24 11:33:02 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 2 probes (2 broadcast(s)) Nov 24 11:33:02 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s) Nov 24 11:33:02 XCP1 iscsi-ha: 26106 Skipping replication_link_check DRBD_LINK_OK=[true] STATE=[master] MANUAL_MODE=[off] Nov 24 11:33:04 XCP1 iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Nov 24 11:33:06 XCP1 ha-lizard: 21935 Spawning new instance of ha-lizard Nov 24 11:33:06 XCP1 ha-lizard: 21935 check_logger_processes: Checking logger processes Nov 24 11:33:06 XCP1 ha-lizard: 21935 check_logger_processes: No processes to clear Nov 24 11:33:07 XCP1 ha-lizard: 26266 LOG_TERMINAL = [false] Nov 24 11:33:07 XCP1 ha-lizard: 26266 Mail Spool Directory Found /dev/shm/ha-lizard-mail Nov 24 11:33:07 XCP1 ha-lizard: 26266 This iteration is count 5 Nov 24 11:33:07 XCP1 ha-lizard: 26266 Checking if this host is a Pool Master or Slave Nov 24 11:33:07 XCP1 ha-lizard: 26266 This host's pool status = master Nov 24 11:33:07 XCP1 ha-lizard: 26266 Checking if ha-lizard is enabled for this pool Nov 24 11:33:07 XCP1 ha-lizard: 26266 check_ha_enabled: Checking if ha-lizard is enabled for pool: f4849b57-cafb-209a-5ebd-82d89b3e9206 Nov 24 11:33:07 XCP1 ha-lizard: 26266 check_ha_enabled: ha-lizard is enabled Nov 24 11:33:07 XCP1 ha-lizard: 26266 check_ha_enabled: checking whether maintenance mode enabled Nov 24 11:33:07 XCP1 ha-lizard: 26266 ha-lizard is enabled Nov 24 11:33:07 XCP1 ha-lizard: 26266 check_xs_ha: Checking XenServer HA status Nov 24 11:33:07 XCP1 ha-lizard: 26266 update_global_conf_params: Successfully updated global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf. Nov 24 11:33:07 XCP1 ha-lizard: 26266 update_global_conf_params: DISABLED_VAPPS=()#012DISK_MONITOR=1#012ENABLE_ALERTS=1#012ENABLE_LOGGING=1#012FENCE_ACTION=stop#012FENCE_ENABLED=1#012FENCE_FILE_LOC=/etc/ha-lizard/fence#012FENCE_HA_ONFAIL=0#012FENCE_HEURISTICS_IPS=10.1.0.3#012FENCE_HOST_FORGET=0#012FENCE_IPADDRESS=#012FENCE_METHOD=POOL#012FENCE_MIN_HOSTS=2#012FENCE_PASSWD=Devildragon.5#012FENCE_QUORUM_REQUIRED=1#012FENCE_REBOOT_LONE_HOST=0#012FENCE_USE_IP_HEURISTICS=1#012GLOBAL_VM_HA=1#012HOST_SELECT_METHOD=0#012MAIL_FROM="root@localhost"#012MAIL_ON=1#012MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME"#012MAIL_TO="root@localhost"#012MGT_LINK_LOSS_TOLERANCE=5#012MONITOR_DELAY=20#012MONITOR_KILLALL=1#012MONITOR_MAX_STARTS=20#012MONITOR_SCANRATE=40#012OP_MODE=2#012PROMOTE_SLAVE=1#012SLAVE_HA=1#012SLAVE_VM_STAT=0#012SMTP_PASS=""#012SMTP_PORT="25"#012SMTP_SERVER="127.0.0.1"#012SMTP_USER=""#012XAPI_COUNT=2#012XAPI_DELAY=10#012XC_FIELD_NAME='ha-lizard-enabled'#012XE_TIMEOUT=30 Nov 24 11:33:07 XCP1 ha-lizard: 26266 check_master_mgt_link_state: Checking management interface link state Nov 24 11:33:07 XCP1 ha-lizard: 26266 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 10.1.0.60 ] Nov 24 11:33:07 XCP1 ha-lizard: 26266 check_master_mgt_link_state: Link [ xapi0 ] state UP Nov 24 11:33:07 XCP1 ha-lizard: 26266 Master management link OK - checking prior link state Nov 24 11:33:07 XCP1 ha-lizard: 26266 This host detected as pool Master Nov 24 11:33:07 XCP1 ha-lizard: 26266 Found 2 hosts in pool Nov 24 11:33:07 XCP1 ha-lizard: 26266 validate_vm_ha_state: Validating VM HA-state Nov 24 11:33:07 XCP1 ha-lizard: 26266 Calling function write_pool_state Nov 24 11:33:07 XCP1 ha-lizard: 26266 Calling function autoselect_slave Nov 24 11:33:07 XCP1 ha-lizard: 26266 Calling function check_slave_status Nov 24 11:33:07 XCP1 ha-lizard: 26266 check_master_mgt_link_state: Checking management interface link state Nov 24 11:33:07 XCP1 ha-lizard: 26266 write_pool_state: MASTER UUID found: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:33:07 XCP1 ha-lizard: 26266 autoselect_slave: This host UUID found: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:33:07 XCP1 ha-lizard: 26266 write_pool_state: MASTER UUID: 4b1d5376-ca55-4417-bf61-c6afa0360dce written to local state storage Nov 24 11:33:08 XCP1 ha-lizard: 26266 autoselect_slave: MASTER host UUID found: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:33:08 XCP1 ha-lizard: 26266 write_pool_state: Calling function get_vms_on_host for UUID: 3a600b7e-2711-4403-9e57-6e0f92aecd38 Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_master_mgt_link_state: Link State = [ true ] for management interface IP [ 10.1.0.60 ] Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_master_mgt_link_state: Link [ xapi0 ] state UP Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_slave_status: Management link OK - continue Nov 24 11:33:08 XCP1 ha-lizard: 26266 get_vms_on_host: No VMs found on host: 3a600b7e-2711-4403-9e57-6e0f92aecd38 Nov 24 11:33:08 XCP1 ha-lizard: 26266 autoselect_slave: 4b1d5376-ca55-4417-bf61-c6afa0360dce is Master UUID - excluding from list of available slaves Nov 24 11:33:08 XCP1 ha-lizard: 26266 write_pool_state: Writing VM array to local state file host.3a600b7e-2711-4403-9e57-6e0f92aecd38.vmlist.uuid_array Nov 24 11:33:08 XCP1 ha-lizard: 26266 autoselect_slave: 1 available Slave UUIDs found: 3a600b7e-2711-4403-9e57-6e0f92aecd38 Nov 24 11:33:08 XCP1 ha-lizard: 26266 write_pool_state: Calling function get_vms_on_host for UUID: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:33:08 XCP1 ha-lizard: 26266 get_pool_host_list: returned 3a600b7e-2711-4403-9e57-6e0f92aecd38#0124b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:33:08 XCP1 ha-lizard: 26266 get_vms_on_host: No VMs found on host: 4b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:33:08 XCP1 ha-lizard: 26266 write_pool_state: Writing VM array to local state file host.4b1d5376-ca55-4417-bf61-c6afa0360dce.vmlist.uuid_array Nov 24 11:33:08 XCP1 ha-lizard: 26266 autoselect_slave: Selected Slave: 3a600b7e-2711-4403-9e57-6e0f92aecd38 = Current slave: 3a600b7e-2711-4403-9e57-6e0f92aecd38 - ignoring update Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_slave_status: Removing Master UUID from list of Hosts Nov 24 11:33:08 XCP1 ha-lizard: 26266 pool autopromote_uuid = [3a600b7e-2711-4403-9e57-6e0f92aecd38] Nov 24 11:33:08 XCP1 ha-lizard: 26266 get_pool_ip_list: returned 10.1.0.61 Nov 24 11:33:08 XCP1 ha-lizard: 26266 write_pool_state: Pool autopromote_uuid=3a600b7e-2711-4403-9e57-6e0f92aecd38 Nov 24 11:33:08 XCP1 ha-lizard: 26266 write_pool_state: autopromote_uuid unchanged - not updating Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_xapi: Pool Host 10.1.0.61 xapi status = 5 Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_ha_enabled: Checking if ha-lizard is enabled for pool: f4849b57-cafb-209a-5ebd-82d89b3e9206 Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_ha_enabled: ha-lizard is enabled Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_ha_enabled: checking whether maintenance mode enabled Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_slave_status: Host [ 3a600b7e-2711-4403-9e57-6e0f92aecd38 ] marked as healthy Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_slave_status: Host IP Address check Status Array for Slaves = (1) Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_slave_status: Quorum check called Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_quorum: Checking host IPs: 10.1.0.61 10.1.0.60 Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_quorum: Host IP: 10.1.0.61 Response = OK Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_quorum: LIVE HOSTs = 1 Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_quorum: Host IP: 10.1.0.60 Response = OK Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_quorum: LIVE HOSTs = 2 Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_quorum: Using network points: 10.1.0.3 as possible additional vote Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_quorum: Heuristic IP: 10.1.0.3 Response = OK Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_quorum: Successful Replies = 1 Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_quorum: Total enpoints checked = 1 with total successful replies = 1 Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_quorum: Additional heuristic vote success. Incremeting vote by 1 Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1 Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_quorum: 3 Hosts found. Minimum needed = 1 + 1. Fencing allowed Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_slave_status: Failed slave count = 0 Nov 24 11:33:08 XCP1 ha-lizard: 26266 check_slave_status: No Failed slaves detected Nov 24 11:33:08 XCP1 ha-lizard: 26266 Function check_slave_status reported no failures: calling vm_mon Nov 24 11:33:08 XCP1 ha-lizard: 26266 write_pool_state: Pool contains 2 hosts. Writing to /etc/ha-lizard/state/pool_num_hosts Nov 24 11:33:08 XCP1 ha-lizard: 26266 get_pool_host_list: enabled flag set - returning only hosts with enabled=true Nov 24 11:33:08 XCP1 ha-lizard: 26266 vm_mon: ha-lizard is operating mode 2 - managing pool VMs Nov 24 11:33:08 XCP1 ha-lizard: 26266 get_pool_host_list: returned 3a600b7e-2711-4403-9e57-6e0f92aecd38#0124b1d5376-ca55-4417-bf61-c6afa0360dce Nov 24 11:33:08 XCP1 ha-lizard: 26266 vm_mon: Retrived list of VMs for this poll: Nov 24 11:33:08 XCP1 ha-lizard: 26266 get_pool_ip_list: returned 10.1.0.61 Nov 24 11:33:08 XCP1 ha-lizard: 26266 vm_mon: Removing Control Domains from VM list Nov 24 11:33:08 XCP1 ha-lizard: 26266 vm_mon: VM list returned = Nov 24 11:33:08 XCP1 ha-lizard: 26266 vm_mon: 0 Eligible Halted VMs found Nov 24 11:33:08 XCP1 ha-lizard: 26266 get_pool_ip_list: returned 10.1.0.61 10.1.0.60 Nov 24 11:33:08 XCP1 ha-lizard: 26266 write_pool_state: Host IP List = 10.1.0.61 10.1.0.60 Nov 24 11:33:08 XCP1 ha-lizard: 26266 write_status_report: Writing status report Nov 24 11:33:09 XCP1 iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Nov 24 11:33:09 XCP1 iscsi-ha: 26090 Spawning new instance of iscsi-ha Nov 24 11:33:09 XCP1 iscsi-ha: 26090 check_logger_processes Checking logger processes Nov 24 11:33:10 XCP1 iscsi-ha: 26090 check_logger_processes No processes to clear Nov 24 11:33:10 XCP1 iscsi-ha: Normalized ISCSI_TARGET_SERVICE [ tgtd ] Nov 24 11:33:10 XCP1 iscsi-ha: XenServer Major Release = [ 8 ] Nov 24 11:33:10 XCP1 iscsi-ha: Mail Spool Directory Found /dev/shm/iscsi-ha-mail Nov 24 11:33:10 XCP1 iscsi-ha: This iteration is count 545 Nov 24 11:33:10 XCP1 iscsi-ha: Checking if this host is a Pool Master or Slave Nov 24 11:33:10 XCP1 iscsi-ha: This host's pool status = master Nov 24 11:33:10 XCP1 iscsi-ha: 26918 service_execute: Execute [ status ] on [ iscsi-ha ] Nov 24 11:33:10 XCP1 iscsi-ha: 26918 service_execute: System V mode detected Nov 24 11:33:10 XCP1 iscsi-ha: auto_plug_pbd: Found LVMoISCSI SR List: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:33:10 XCP1 iscsi-ha: 26918 service_execute: [ OK ]#015iscsi-ha running: 3968 Nov 24 11:33:10 XCP1 iscsi-ha: 26918 service_execute: Returning exit status [ 0 ] Nov 24 11:33:10 XCP1 iscsi-ha: 26918 DRBD Running on this host: version: 8.4.10 (api:1/proto:86-101) srcversion: CE41F2FAB70AE3534BF40A6 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- ns:158981456 nr:0 dw:10372 dr:158974480 al:2 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:171667012 [========>...........] sync'ed: 48.1% (167640/322888)M finish: 1:39:29 speed: 28,752 (28,844) K/sec Nov 24 11:33:10 XCP1 iscsi-ha: 26918 validate_drbd_resources_loaded: Checking DRBD has loaded with resources. Checking [ 7 ] > [ 2 ] Nov 24 11:33:10 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:33:10 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:33:10 XCP1 iscsi-ha: 26918 validate_drbd_resources_loaded: Resources loaded Nov 24 11:33:10 XCP1 iscsi-ha: 26918 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode Nov 24 11:33:10 XCP1 iscsi-ha: 26918 DRBD resource: iscsi1 is currently syncing [SyncSource] Nov 24 11:33:10 XCP1 iscsi-ha: 26918 service_execute: Execute [ status ] on [ tgtd ] Nov 24 11:33:10 XCP1 iscsi-ha: 26918 service_execute: systemctl mode being used Nov 24 11:33:10 XCP1 iscsi-ha: 26918 service_execute: ? tgtd.service - tgtd iSCSI target daemon#012 Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012 Drop-In: /etc/systemd/system/tgtd.service.d#012 L¦local.conf#012 Active: active (running) since Tue 2020-11-24 10:01:23 CET; 1h 31min ago#012 Process: 4859 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)#012 Process: 4827 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)#012 Process: 4825 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)#012 Process: 4734 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS)#012 Main PID: 4733 (tgtd)#012 CGroup: /system.slice/tgtd.service#012 L¦4733 /usr/sbin/tgtd -f Nov 24 11:33:10 XCP1 iscsi-ha: 26918 service_execute: Returning exit status [ 0 ] Nov 24 11:33:10 XCP1 iscsi-ha: 26918 iSCSI target: tgtd status = OK. [ active (running) since Tue 2020-11-24 10:01:23 CET; 1h 31min ago ] Nov 24 11:33:10 XCP1 iscsi-ha: 26918 local_ip_list: Local IP list returned 127.0.0.1#01210.1.0.60#01210.1.10.2#01210.1.10.4 Nov 24 11:33:10 XCP1 iscsi-ha: 26918 CHECKING IP 127.0.0.1 Nov 24 11:33:10 XCP1 iscsi-ha: 26918 CHECKING IP 10.1.0.60 Nov 24 11:33:10 XCP1 iscsi-ha: 26918 CHECKING IP 10.1.10.2 Nov 24 11:33:10 XCP1 iscsi-ha: 26918 CHECKING IP 10.1.10.4 Nov 24 11:33:10 XCP1 iscsi-ha: 26918 Virtual IP: 10.1.10.4 discovered on host XCP1 Nov 24 11:33:10 XCP1 iscsi-ha: 26918 send_replication_network_arp: Sending ARP update to peer Nov 24 11:33:10 XCP1 iscsi-ha: 26918 send_replication_network_arp: IP address list for [ xapi1 ] = [ 10.1.10.2#01210.1.10.4 ] Nov 24 11:33:10 XCP1 iscsi-ha: 26918 send_replication_network_arp: Updating ARP for device [ xapi1 ] IP [ 10.1.10.2 ] Nov 24 11:33:11 XCP1 ha-lizard: 1843 ha-lizard Watchdog: ha-lizard running - OK Nov 24 11:33:11 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.1.10.2 from 10.1.10.2 xapi1 Nov 24 11:33:11 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 2 probes (2 broadcast(s)) Nov 24 11:33:11 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s) Nov 24 11:33:11 XCP1 iscsi-ha: 26918 send_replication_network_arp: Updating ARP for device [ xapi1 ] IP [ 10.1.10.4 ] Nov 24 11:33:12 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.1.10.4 from 10.1.10.4 xapi1 Nov 24 11:33:12 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 2 probes (2 broadcast(s)) Nov 24 11:33:12 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s) Nov 24 11:33:12 XCP1 iscsi-ha: 26918 Skipping replication_link_check DRBD_LINK_OK=[true] STATE=[master] MANUAL_MODE=[off] Nov 24 11:33:14 XCP1 iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Nov 24 11:33:19 XCP1 iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Nov 24 11:33:20 XCP1 iscsi-ha: 26900 Spawning new instance of iscsi-ha Nov 24 11:33:20 XCP1 iscsi-ha: 26900 check_logger_processes Checking logger processes Nov 24 11:33:20 XCP1 iscsi-ha: 26900 check_logger_processes No processes to clear Nov 24 11:33:20 XCP1 iscsi-ha: Normalized ISCSI_TARGET_SERVICE [ tgtd ] Nov 24 11:33:20 XCP1 iscsi-ha: XenServer Major Release = [ 8 ] Nov 24 11:33:20 XCP1 iscsi-ha: Mail Spool Directory Found /dev/shm/iscsi-ha-mail Nov 24 11:33:20 XCP1 iscsi-ha: This iteration is count 546 Nov 24 11:33:20 XCP1 iscsi-ha: Checking if this host is a Pool Master or Slave Nov 24 11:33:20 XCP1 iscsi-ha: This host's pool status = master Nov 24 11:33:20 XCP1 iscsi-ha: 27177 service_execute: Execute [ status ] on [ iscsi-ha ] Nov 24 11:33:20 XCP1 iscsi-ha: 27177 service_execute: System V mode detected Nov 24 11:33:20 XCP1 iscsi-ha: auto_plug_pbd: Found LVMoISCSI SR List: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:33:20 XCP1 iscsi-ha: 27177 service_execute: [ OK ]#015iscsi-ha running: 3968 Nov 24 11:33:20 XCP1 iscsi-ha: 27177 service_execute: Returning exit status [ 0 ] Nov 24 11:33:20 XCP1 iscsi-ha: 27177 DRBD Running on this host: version: 8.4.10 (api:1/proto:86-101) srcversion: CE41F2FAB70AE3534BF40A6 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----- ns:159272768 nr:0 dw:10372 dr:159265792 al:2 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:171375700 [========>...........] sync'ed: 48.2% (167356/322888)M finish: 1:37:06 speed: 29,396 (28,840) K/sec Nov 24 11:33:20 XCP1 iscsi-ha: 27177 validate_drbd_resources_loaded: Checking DRBD has loaded with resources. Checking [ 7 ] > [ 2 ] Nov 24 11:33:20 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Scanning for Volume Group -> iscsi-sr: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:33:20 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Volume Group for iSCSI-SR found OK: 3e3a1c32-19ff-3400-f2b1-0a5cf5fca4f6 Nov 24 11:33:20 XCP1 iscsi-ha: 27177 validate_drbd_resources_loaded: Resources loaded Nov 24 11:33:20 XCP1 iscsi-ha: 27177 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode Nov 24 11:33:20 XCP1 iscsi-ha: 27177 DRBD resource: iscsi1 is currently syncing [SyncSource] Nov 24 11:33:20 XCP1 iscsi-ha: 27177 service_execute: Execute [ status ] on [ tgtd ] Nov 24 11:33:20 XCP1 iscsi-ha: 27177 service_execute: systemctl mode being used Nov 24 11:33:20 XCP1 iscsi-ha: 27177 service_execute: ? tgtd.service - tgtd iSCSI target daemon#012 Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012 Drop-In: /etc/systemd/system/tgtd.service.d#012 L¦local.conf#012 Active: active (running) since Tue 2020-11-24 10:01:23 CET; 1h 31min ago#012 Process: 4859 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)#012 Process: 4827 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)#012 Process: 4825 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)#012 Process: 4734 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS)#012 Main PID: 4733 (tgtd)#012 CGroup: /system.slice/tgtd.service#012 L¦4733 /usr/sbin/tgtd -f Nov 24 11:33:20 XCP1 iscsi-ha: 27177 service_execute: Returning exit status [ 0 ] Nov 24 11:33:20 XCP1 iscsi-ha: 27177 iSCSI target: tgtd status = OK. [ active (running) since Tue 2020-11-24 10:01:23 CET; 1h 31min ago ] Nov 24 11:33:20 XCP1 iscsi-ha: 27177 local_ip_list: Local IP list returned 127.0.0.1#01210.1.0.60#01210.1.10.2#01210.1.10.4 Nov 24 11:33:20 XCP1 iscsi-ha: 27177 CHECKING IP 127.0.0.1 Nov 24 11:33:20 XCP1 iscsi-ha: 27177 CHECKING IP 10.1.0.60 Nov 24 11:33:20 XCP1 iscsi-ha: 27177 CHECKING IP 10.1.10.2 Nov 24 11:33:20 XCP1 iscsi-ha: 27177 CHECKING IP 10.1.10.4 Nov 24 11:33:20 XCP1 iscsi-ha: 27177 Virtual IP: 10.1.10.4 discovered on host XCP1 Nov 24 11:33:20 XCP1 iscsi-ha: 27177 send_replication_network_arp: Sending ARP update to peer Nov 24 11:33:20 XCP1 iscsi-ha: 27177 send_replication_network_arp: IP address list for [ xapi1 ] = [ 10.1.10.2#01210.1.10.4 ] Nov 24 11:33:20 XCP1 iscsi-ha: 27177 send_replication_network_arp: Updating ARP for device [ xapi1 ] IP [ 10.1.10.2 ] Nov 24 11:33:21 XCP1 ha-lizard: 1843 ha-lizard Watchdog: ha-lizard running - OK Nov 24 11:33:21 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.1.10.2 from 10.1.10.2 xapi1 Nov 24 11:33:21 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 2 probes (2 broadcast(s)) Nov 24 11:33:21 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Received 0 response(s) Nov 24 11:33:21 XCP1 iscsi-ha: 27177 send_replication_network_arp: Updating ARP for device [ xapi1 ] IP [ 10.1.10.4 ] Nov 24 11:33:22 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: ARPING 10.1.10.4 from 10.1.10.4 xapi1 Nov 24 11:33:22 XCP1 iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Sent 2 probes (2 broadcast(s))