Sep 21 13:29:27 ch-pool2022b iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Pool Master 172.168.1.16 not responding - replug_pbd exiting Sep 21 13:29:27 ch-pool2022b iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Sep 21 13:29:28 ch-pool2022b ha-lizard: 13732 xe_wrapper: COMMAND: xe pool-param-get Sep 21 13:29:28 ch-pool2022b ha-lizard: 13732 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 21 13:29:28 ch-pool2022b ha-lizard: 13732 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Sep 21 13:29:28 ch-pool2022b ha-lizard: 13732 FLUSH_MAIL_EXEC returned [0] Sep 21 13:29:28 ch-pool2022b ha-lizard: 13732 check_email_enabled: Email enabled for xe_wrapper Sep 21 13:29:28 ch-pool2022b ha-lizard: 13732 email: Sending ALERT email to myemail@myemail.com: xe_wrapper: COMMAND: xe pool-param-get Sep 21 13:29:28 ch-pool2022b ha-lizard: 13732 get_alert_level: Fetching alert level for [xe_wrapper] Sep 21 13:29:28 ch-pool2022b ha-lizard: 13732 xe_raise_alert: Raising system alert [HA-Lizard - xe_wrapper] with message [xe_wrapper: COMMAND: xe pool-param-get] severity [1] Sep 21 13:29:32 ch-pool2022b ha-lizard: 13732 service_execute: Execute [ stop ] on [ ha-lizard ] Sep 21 13:29:32 ch-pool2022b ha-lizard: 13732 service_execute: System V mode detected Sep 21 13:29:32 ch-pool2022b ha-lizard: 13732 service_execute: Stopping ha-lizard (via systemctl): [ OK ] Sep 21 13:29:32 ch-pool2022b ha-lizard: 13732 service_execute: Returning exit status [ 0 ] Sep 21 13:29:32 ch-pool2022b ha-lizard: 13732 pool autopromote_uuid = [Stopping ha-lizard (via systemctl): [ OK ]#015] Sep 21 13:29:32 ch-pool2022b iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Sep 21 13:29:34 ch-pool2022b ha-lizard: 2413 ha-lizard Watchdog: ha-lizard is stopped, starting ha-lizard Sep 21 13:29:34 ch-pool2022b ha-lizard-init: Waiting for xapi initialize Sep 21 13:29:34 ch-pool2022b ha-lizard-init: XAPI initialized on attempt [ 0 ] Sep 21 13:29:34 ch-pool2022b ha-lizard: 18288 Spawning new instance of ha-lizard Sep 21 13:29:34 ch-pool2022b ha-lizard: 18288 check_logger_processes: Checking logger processes Sep 21 13:29:34 ch-pool2022b ha-lizard: 18288 check_logger_processes: No processes to clear Sep 21 13:29:34 ch-pool2022b ha-lizard: 18332 LOG_TERMINAL = [false] Sep 21 13:29:34 ch-pool2022b ha-lizard: 18332 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 21 13:29:34 ch-pool2022b ha-lizard: 18332 This iteration is count 1 Sep 21 13:29:34 ch-pool2022b ha-lizard: 18332 Checking if this host is a Pool Master or Slave Sep 21 13:29:34 ch-pool2022b ha-lizard: 18332 This host's pool status = slave:172.168.1.16 Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18030 Spawning new instance of iscsi-ha Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18030 check_logger_processes Checking logger processes Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18030 check_logger_processes No processes to clear Sep 21 13:29:36 ch-pool2022b iscsi-ha: Normalized ISCSI_TARGET_SERVICE [ tgtd ] Sep 21 13:29:36 ch-pool2022b iscsi-ha: XenServer Major Release = [ 8 ] Sep 21 13:29:36 ch-pool2022b iscsi-ha: Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 21 13:29:36 ch-pool2022b iscsi-ha: This iteration is count 952 Sep 21 13:29:36 ch-pool2022b iscsi-ha: Checking if this host is a Pool Master or Slave Sep 21 13:29:36 ch-pool2022b iscsi-ha: This host's pool status = slave:172.168.1.16 Sep 21 13:29:36 ch-pool2022b iscsi-ha: service_execute: Execute [ status ] on [ iscsi-ha ] Sep 21 13:29:36 ch-pool2022b iscsi-ha: service_execute: System V mode detected Sep 21 13:29:36 ch-pool2022b iscsi-ha: service_execute: [ OK ]#015iscsi-ha running: 10849 Sep 21 13:29:36 ch-pool2022b iscsi-ha: service_execute: Returning exit status [ 0 ] Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 local_ip_list: Local IP list returned 127.0.0.1#01210.10.10.2#012172.168.1.17 Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 service_execute: Execute [ status ] on [ tgtd ] Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 service_execute: systemctl mode being used Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012 Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012 Drop-In: /etc/systemd/system/tgtd.service.d#012 └─local.conf#012 Active: failed (Result: signal) since Wed 2022-09-21 11:24:23 MDT; 2h 5min ago#012 Process: 12370 ExecStop=/usr/sbin/tgtadm --op delete --mode system (code=exited, status=22)#012 Process: 12353 ExecStop=/usr/sbin/tgt-admin --update ALL -c /dev/null (code=exited, status=0/SUCCESS)#012 Process: 12349 ExecStop=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)#012 Process: 11683 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)#012 Process: 11651 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)#012 Process: 11649 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)#012 Process: 11553 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS)#012 Process: 11552 ExecStart=/usr/sbin/tgtd -f $TGTD_OPTS (code=killed, signal=KILL)#012 Main PID: 11552 (code=killed, signal=KILL) Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 service_execute: Returning exit status [ 3 ] Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 iSCSI target: tgtd status stopped. Expected Stopped . [failed (Result: signal) since Wed 2022-09-21 11:24:23 MDT; 2h 5min ago] Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 DRBD Running on this host: version: 8.4.10 (api:1/proto:86-101)#012srcversion: CE41F2FAB70AE3534BF40A6 #012#012 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----#012 ns:0 nr:4100 dw:4100 dr:672 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 validate_drbd_resources_loaded: Checking DRBD has loaded with resources. Checking [ 5 ] > [ 2 ] Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 validate_drbd_resources_loaded: Resources loaded Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 DRBD Resource: [iscsi1] in [WFConnection] state - expected Connected state Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 email: Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 email: Duplicate message - not sending. Content = DRBD Resource: [iscsi1] in [WFConnection] state - expected Connected state Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 email: Message barred for 30 minutes Sep 21 13:29:36 ch-pool2022b iscsi-ha: 18422 Skipping replication_link_check DRBD_LINK_OK=[false] STATE=[slave:172.168.1.16] MANUAL_MODE=[off] Sep 21 13:29:37 ch-pool2022b iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Pool Master 172.168.1.16 not responding - replug_pbd exiting Sep 21 13:29:37 ch-pool2022b iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Sep 21 13:29:40 ch-pool2022b ha-lizard: 18332 update_global_conf_params: Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 21 13:29:40 ch-pool2022b ha-lizard: 18332 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 21 13:29:40 ch-pool2022b ha-lizard: 18332 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Sep 21 13:29:40 ch-pool2022b ha-lizard: 18332 FLUSH_MAIL_EXEC returned [0] Sep 21 13:29:40 ch-pool2022b ha-lizard: 18332 check_email_enabled: Email enabled for update_global_conf_params Sep 21 13:29:40 ch-pool2022b ha-lizard: 18332 email: Sending ALERT email to myemail@myemail.com: update_global_conf_params: Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 21 13:29:40 ch-pool2022b ha-lizard: 18332 get_alert_level: Fetching alert level for [update_global_conf_params] Sep 21 13:29:41 ch-pool2022b ha-lizard: 18332 xe_raise_alert: Raising system alert [HA-Lizard - update_global_conf_params] with message [update_global_conf_params: Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration!] severity [3] Sep 21 13:29:42 ch-pool2022b ha-lizard: 18332 master_ip: Pool Master IP Address = 172.168.1.16 Sep 21 13:29:42 ch-pool2022b ha-lizard: 18332 Validating master is still a master Sep 21 13:29:42 ch-pool2022b ha-lizard: 18332 [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.168.1.16 ] Sep 21 13:29:42 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Traceback (most recent call last): Sep 21 13:29:42 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: File "/etc/ha-lizard/scripts/host_is_slave", line 35, in Sep 21 13:29:42 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: session.xenapi.login_with_password('root','') Sep 21 13:29:42 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: File "/usr/lib/python2.7/site-packages/XenAPI.py", line 264, in __call__ Sep 21 13:29:42 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: return self.__send(self.__name, args) Sep 21 13:29:42 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: File "/usr/lib/python2.7/site-packages/XenAPI.py", line 151, in xenapi_request Sep 21 13:29:42 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: self._login(methodname, params) Sep 21 13:29:42 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: File "/usr/lib/python2.7/site-packages/XenAPI.py", line 188, in _login Sep 21 13:29:42 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: raise e Sep 21 13:29:42 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: socket.error: [Errno 113] No route to host Sep 21 13:29:42 ch-pool2022b ha-lizard: 18332 This slave - ch-pool2022b: dd897c2a-f2d9-4aed-ae33-21d89938d849 selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1 Sep 21 13:29:42 ch-pool2022b iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Sep 21 13:29:43 ch-pool2022b ha-lizard: 18332 check_xapi: Pool Host 172.168.1.16 failed to respond to ICMP ping Sep 21 13:29:43 ch-pool2022b ha-lizard: 18332 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 21 13:29:43 ch-pool2022b ha-lizard: 18332 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Sep 21 13:29:43 ch-pool2022b ha-lizard: 18332 FLUSH_MAIL_EXEC returned [0] Sep 21 13:29:43 ch-pool2022b ha-lizard: 18332 check_email_enabled: Email enabled for check_xapi Sep 21 13:29:43 ch-pool2022b ha-lizard: 18332 email: Sending ALERT email to myemail@myemail.com: check_xapi: Pool Host on Server: 172.168.1.16 not responding to ICMP ping - manual intervention may be required Sep 21 13:29:43 ch-pool2022b ha-lizard: 18332 get_alert_level: Fetching alert level for [check_xapi] Sep 21 13:29:43 ch-pool2022b ha-lizard: 18332 xe_raise_alert: Raising system alert [HA-Lizard - check_xapi] with message [check_xapi: Pool Host on Server: 172.168.1.16 not responding to ICMP ping - manual intervention may be required] severity [1] Sep 21 13:29:44 ch-pool2022b ha-lizard: 18332 Pool Master NOT OK - Checking if ha-lizard is enabled in latest state file Sep 21 13:29:44 ch-pool2022b ha-lizard: 18332 Checking if ha-lizard is enabled Sep 21 13:29:44 ch-pool2022b ha-lizard: 18332 Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled Sep 21 13:29:44 ch-pool2022b ha-lizard: 18332 ha-lizard is disabled - exiting Sep 21 13:29:44 ch-pool2022b ha-lizard: 2413 ha-lizard Watchdog: ha-lizard running - OK Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18387 Spawning new instance of iscsi-ha Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18387 check_logger_processes Checking logger processes Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18387 check_logger_processes No processes to clear Sep 21 13:29:46 ch-pool2022b iscsi-ha: Normalized ISCSI_TARGET_SERVICE [ tgtd ] Sep 21 13:29:46 ch-pool2022b iscsi-ha: XenServer Major Release = [ 8 ] Sep 21 13:29:46 ch-pool2022b iscsi-ha: Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 21 13:29:46 ch-pool2022b iscsi-ha: This iteration is count 953 Sep 21 13:29:46 ch-pool2022b iscsi-ha: Checking if this host is a Pool Master or Slave Sep 21 13:29:46 ch-pool2022b iscsi-ha: This host's pool status = slave:172.168.1.16 Sep 21 13:29:46 ch-pool2022b iscsi-ha: service_execute: Execute [ status ] on [ iscsi-ha ] Sep 21 13:29:46 ch-pool2022b iscsi-ha: service_execute: System V mode detected Sep 21 13:29:46 ch-pool2022b iscsi-ha: service_execute: [ OK ]#015iscsi-ha running: 10849 Sep 21 13:29:46 ch-pool2022b iscsi-ha: service_execute: Returning exit status [ 0 ] Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 local_ip_list: Local IP list returned 127.0.0.1#01210.10.10.2#012172.168.1.17 Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 service_execute: Execute [ status ] on [ tgtd ] Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 service_execute: systemctl mode being used Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012 Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012 Drop-In: /etc/systemd/system/tgtd.service.d#012 └─local.conf#012 Active: failed (Result: signal) since Wed 2022-09-21 11:24:23 MDT; 2h 5min ago#012 Process: 12370 ExecStop=/usr/sbin/tgtadm --op delete --mode system (code=exited, status=22)#012 Process: 12353 ExecStop=/usr/sbin/tgt-admin --update ALL -c /dev/null (code=exited, status=0/SUCCESS)#012 Process: 12349 ExecStop=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)#012 Process: 11683 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)#012 Process: 11651 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)#012 Process: 11649 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)#012 Process: 11553 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS)#012 Process: 11552 ExecStart=/usr/sbin/tgtd -f $TGTD_OPTS (code=killed, signal=KILL)#012 Main PID: 11552 (code=killed, signal=KILL) Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 service_execute: Returning exit status [ 3 ] Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 iSCSI target: tgtd status stopped. Expected Stopped . [failed (Result: signal) since Wed 2022-09-21 11:24:23 MDT; 2h 5min ago] Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 DRBD Running on this host: version: 8.4.10 (api:1/proto:86-101)#012srcversion: CE41F2FAB70AE3534BF40A6 #012#012 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----#012 ns:0 nr:4100 dw:4100 dr:672 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 validate_drbd_resources_loaded: Checking DRBD has loaded with resources. Checking [ 5 ] > [ 2 ] Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 validate_drbd_resources_loaded: Resources loaded Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 DRBD Resource: [iscsi1] in [WFConnection] state - expected Connected state Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 email: Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 email: Duplicate message - not sending. Content = DRBD Resource: [iscsi1] in [WFConnection] state - expected Connected state Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 email: Message barred for 30 minutes Sep 21 13:29:46 ch-pool2022b iscsi-ha: 18671 Skipping replication_link_check DRBD_LINK_OK=[false] STATE=[slave:172.168.1.16] MANUAL_MODE=[off] Sep 21 13:29:47 ch-pool2022b iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Pool Master 172.168.1.16 not responding - replug_pbd exiting Sep 21 13:29:47 ch-pool2022b iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Sep 21 13:29:49 ch-pool2022b ha-lizard: 18288 Spawning new instance of ha-lizard Sep 21 13:29:49 ch-pool2022b ha-lizard: 18288 check_logger_processes: Checking logger processes Sep 21 13:29:49 ch-pool2022b ha-lizard: 18288 check_logger_processes: No processes to clear Sep 21 13:29:49 ch-pool2022b ha-lizard: 18772 LOG_TERMINAL = [false] Sep 21 13:29:49 ch-pool2022b ha-lizard: 18772 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 21 13:29:49 ch-pool2022b ha-lizard: 18772 This iteration is count 2 Sep 21 13:29:49 ch-pool2022b ha-lizard: 18772 Checking if this host is a Pool Master or Slave Sep 21 13:29:49 ch-pool2022b ha-lizard: 18772 This host's pool status = slave:172.168.1.16 Sep 21 13:29:52 ch-pool2022b iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Sep 21 13:29:54 ch-pool2022b ha-lizard: 2413 ha-lizard Watchdog: ha-lizard running - OK Sep 21 13:29:56 ch-pool2022b ha-lizard: 18772 update_global_conf_params: Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 21 13:29:56 ch-pool2022b ha-lizard: 18772 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 21 13:29:56 ch-pool2022b ha-lizard: 18772 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Sep 21 13:29:56 ch-pool2022b ha-lizard: 18772 FLUSH_MAIL_EXEC returned [0] Sep 21 13:29:56 ch-pool2022b ha-lizard: 18772 check_email_enabled: Email enabled for update_global_conf_params Sep 21 13:29:56 ch-pool2022b ha-lizard: 18772 email: Sending ALERT email to myemail@myemail.com: update_global_conf_params: Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration! Sep 21 13:29:56 ch-pool2022b ha-lizard: 18772 get_alert_level: Fetching alert level for [update_global_conf_params] Sep 21 13:29:56 ch-pool2022b ha-lizard: 18772 xe_raise_alert: Raising system alert [HA-Lizard - update_global_conf_params] with message [update_global_conf_params: Failed to update global pool configuration settings in /etc/ha-lizard/ha-lizard.pool.conf - Check Configuration!] severity [3] Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18636 Spawning new instance of iscsi-ha Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18636 check_logger_processes Checking logger processes Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18636 check_logger_processes No processes to clear Sep 21 13:29:56 ch-pool2022b iscsi-ha: Normalized ISCSI_TARGET_SERVICE [ tgtd ] Sep 21 13:29:56 ch-pool2022b iscsi-ha: XenServer Major Release = [ 8 ] Sep 21 13:29:56 ch-pool2022b iscsi-ha: Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 21 13:29:56 ch-pool2022b iscsi-ha: This iteration is count 954 Sep 21 13:29:56 ch-pool2022b iscsi-ha: Checking if this host is a Pool Master or Slave Sep 21 13:29:56 ch-pool2022b iscsi-ha: This host's pool status = slave:172.168.1.16 Sep 21 13:29:56 ch-pool2022b iscsi-ha: service_execute: Execute [ status ] on [ iscsi-ha ] Sep 21 13:29:56 ch-pool2022b iscsi-ha: service_execute: System V mode detected Sep 21 13:29:56 ch-pool2022b iscsi-ha: service_execute: [ OK ]#015iscsi-ha running: 10849 Sep 21 13:29:56 ch-pool2022b iscsi-ha: service_execute: Returning exit status [ 0 ] Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 local_ip_list: Local IP list returned 127.0.0.1#01210.10.10.2#012172.168.1.17 Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 service_execute: Execute [ status ] on [ tgtd ] Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 service_execute: systemctl mode being used Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 service_execute: ● tgtd.service - tgtd iSCSI target daemon#012 Loaded: loaded (/usr/lib/systemd/system/tgtd.service; disabled; vendor preset: disabled)#012 Drop-In: /etc/systemd/system/tgtd.service.d#012 └─local.conf#012 Active: failed (Result: signal) since Wed 2022-09-21 11:24:23 MDT; 2h 5min ago#012 Process: 12370 ExecStop=/usr/sbin/tgtadm --op delete --mode system (code=exited, status=22)#012 Process: 12353 ExecStop=/usr/sbin/tgt-admin --update ALL -c /dev/null (code=exited, status=0/SUCCESS)#012 Process: 12349 ExecStop=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)#012 Process: 11683 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS)#012 Process: 11651 ExecStartPost=/usr/sbin/tgt-admin -e -c $TGTD_CONFIG (code=exited, status=0/SUCCESS)#012 Process: 11649 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS)#012 Process: 11553 ExecStartPost=/bin/sleep 5 (code=exited, status=0/SUCCESS)#012 Process: 11552 ExecStart=/usr/sbin/tgtd -f $TGTD_OPTS (code=killed, signal=KILL)#012 Main PID: 11552 (code=killed, signal=KILL) Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 service_execute: Returning exit status [ 3 ] Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 iSCSI target: tgtd status stopped. Expected Stopped . [failed (Result: signal) since Wed 2022-09-21 11:24:23 MDT; 2h 5min ago] Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 DRBD Running on this host: version: 8.4.10 (api:1/proto:86-101)#012srcversion: CE41F2FAB70AE3534BF40A6 #012#012 1: cs:WFConnection ro:Secondary/Unknown ds:UpToDate/DUnknown C r-----#012 ns:0 nr:4100 dw:4100 dr:672 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 validate_drbd_resources_loaded: Checking DRBD has loaded with resources. Checking [ 5 ] > [ 2 ] Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 validate_drbd_resources_loaded: Resources loaded Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 DRBD Resource: [iscsi1] in [WFConnection] state - expected Connected state Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 email: Mail Spool Directory Found /dev/shm/iscsi-ha-mail Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 email: Duplicate message - not sending. Content = DRBD Resource: [iscsi1] in [WFConnection] state - expected Connected state Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 email: Message barred for 30 minutes Sep 21 13:29:56 ch-pool2022b iscsi-ha: 18926 Skipping replication_link_check DRBD_LINK_OK=[false] STATE=[slave:172.168.1.16] MANUAL_MODE=[off] Sep 21 13:29:57 ch-pool2022b ha-lizard: 18772 master_ip: Pool Master IP Address = 172.168.1.16 Sep 21 13:29:57 ch-pool2022b ha-lizard: 18772 Validating master is still a master Sep 21 13:29:57 ch-pool2022b ha-lizard: 18772 [ /etc/ha-lizard/scripts/timeout 1 /etc/ha-lizard/scripts/host_is_slave 172.168.1.16 ] Sep 21 13:29:57 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: Traceback (most recent call last): Sep 21 13:29:57 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: File "/etc/ha-lizard/scripts/host_is_slave", line 35, in Sep 21 13:29:57 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: session.xenapi.login_with_password('root','') Sep 21 13:29:57 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: File "/usr/lib/python2.7/site-packages/XenAPI.py", line 264, in __call__ Sep 21 13:29:57 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: return self.__send(self.__name, args) Sep 21 13:29:57 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: File "/usr/lib/python2.7/site-packages/XenAPI.py", line 151, in xenapi_request Sep 21 13:29:57 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: self._login(methodname, params) Sep 21 13:29:57 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: File "/usr/lib/python2.7/site-packages/XenAPI.py", line 188, in _login Sep 21 13:29:57 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: raise e Sep 21 13:29:57 ch-pool2022b ha-lizard-ERROR-/etc/ha-lizard/init/ha-lizard.mon: socket.error: [Errno 113] No route to host Sep 21 13:29:57 ch-pool2022b iscsi-ha-NOTICE-/etc/iscsi-ha/init/iscsi-ha.mon: Pool Master 172.168.1.16 not responding - replug_pbd exiting Sep 21 13:29:57 ch-pool2022b ha-lizard: 18772 This slave - ch-pool2022b: dd897c2a-f2d9-4aed-ae33-21d89938d849 selected as allowed to become master: setting ALLOW_PROMOTE_MASTER=1 Sep 21 13:29:57 ch-pool2022b iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Sep 21 13:29:58 ch-pool2022b ha-lizard: 18772 check_xapi: Pool Host 172.168.1.16 failed to respond to ICMP ping Sep 21 13:29:58 ch-pool2022b ha-lizard: 18772 email: Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 21 13:29:58 ch-pool2022b ha-lizard: 18772 Ready to exec [find /dev/shm/ha-lizard-mail/ -name *.msg -type f -mmin +60 -delete] Sep 21 13:29:58 ch-pool2022b ha-lizard: 18772 FLUSH_MAIL_EXEC returned [0] Sep 21 13:29:58 ch-pool2022b ha-lizard: 18772 check_email_enabled: Email enabled for check_xapi Sep 21 13:29:58 ch-pool2022b ha-lizard: 18772 email: Sending ALERT email to myemail@myemail.com: check_xapi: Pool Host on Server: 172.168.1.16 not responding to ICMP ping - manual intervention may be required Sep 21 13:29:58 ch-pool2022b ha-lizard: 18772 get_alert_level: Fetching alert level for [check_xapi] Sep 21 13:29:58 ch-pool2022b ha-lizard: 18772 xe_raise_alert: Raising system alert [HA-Lizard - check_xapi] with message [check_xapi: Pool Host on Server: 172.168.1.16 not responding to ICMP ping - manual intervention may be required] severity [1] Sep 21 13:29:59 ch-pool2022b ha-lizard: 18772 Pool Master NOT OK - Checking if ha-lizard is enabled in latest state file Sep 21 13:29:59 ch-pool2022b ha-lizard: 18772 Checking if ha-lizard is enabled Sep 21 13:29:59 ch-pool2022b ha-lizard: 18772 Statefile /etc/ha-lizard/state/ha_lizard_enabled found: checking if ha-lizard is enabled Sep 21 13:29:59 ch-pool2022b ha-lizard: 18772 ha-lizard is disabled - exiting Sep 21 13:30:02 ch-pool2022b iscsi-ha: iscsi-ha Watchdog: iscsi-ha running - OK Sep 21 13:30:04 ch-pool2022b ha-lizard: 2413 ha-lizard Watchdog: ha-lizard running - OK Sep 21 13:30:04 ch-pool2022b ha-lizard: 18288 Spawning new instance of ha-lizard Sep 21 13:30:04 ch-pool2022b ha-lizard: 18288 check_logger_processes: Checking logger processes Sep 21 13:30:04 ch-pool2022b ha-lizard: 18288 check_logger_processes: No processes to clear Sep 21 13:30:05 ch-pool2022b ha-lizard: 19136 LOG_TERMINAL = [false] Sep 21 13:30:05 ch-pool2022b ha-lizard: 19136 Mail Spool Directory Found /dev/shm/ha-lizard-mail Sep 21 13:30:05 ch-pool2022b ha-lizard: 19136 This iteration is count 3 Sep 21 13:30:05 ch-pool2022b ha-lizard: 19136 Checking if this host is a Pool Master or Slave Sep 21 13:30:05 ch-pool2022b ha-lizard: 19136 This host's pool status = slave:172.168.1.16