TOPIC:

I have a 2 node pool built and running. 1 iscsi SR. two win2k8 VMs

So i built everything and had it running for a week with total stability. So this week i have been doing some DR testing. I had both vm's running on the slave host and pulled the plug on the slave host figuring that i would see both vm's migrate over to the master. This did not happen. Upon rebooting the slave. The vm's did then migrate over to the master and boot back up. The slave self fenced and i did run the recover_fenced_host script and ha-cfg status shows that it is now enabled and runing. service ha-lizard status also show its running.

Currently xencenter is reporting that the ISCSI is connected to the master but unplugged from the slave. performing a repair in xencenter doesn't fix it. if i do a iscsi-cfg status command on the slave the ISCSI TARGET reports that tgtd is stopped. if i do a service tgtd start command i get prompted that it exited with code: 22.

please help

It is correct to see TGTD stopped on the slave. It should be running on the master only. This is controlled by iscsi-ha.

Please send your configuration for halizard (ha-cfg get) and also iscsi-ha.

[root@HA-xen02 ~]# ha-cfg get
DISABLED_VAPPS=() FENCE_IPADDRESS= MAIL_FROM="root@localhost" OP_MODE=2
ENABLE_LOGGING=1 FENCE_METHOD=POOL MAIL_ON=1 PROMOTE_SLAVE=1
FENCE_ACTION=stop FENCE_MIN_HOSTS=2 MAIL_SUBJECT="SYSTEM_ALERT-FROM_HOST:$HOSTNAME" SLAVE_HA=1
FENCE_ENABLED=1 FENCE_PASSWD= MAIL_TO=****** SLAVE_VM_STAT=0
FENCE_FILE_LOC=/etc/ha-lizard/fence FENCE_QUORUM_REQUIRED=1 MONITOR_DELAY=15 XAPI_COUNT=2
FENCE_HA_ONFAIL=0 FENCE_REBOOT_LONE_HOST=0 MONITOR_KILLALL=1 XAPI_DELAY=10
FENCE_HEURISTICS_IPS=192.168.1.253 FENCE_USE_IP_HEURISTICS=1 MONITOR_MAX_STARTS=20 XC_FIELD_NAME='ha-lizard-enabled'
FENCE_HOST_FORGET=0 GLOBAL_VM_HA=1 MONITOR_SCANRATE=10 XE_TIMEOUT=10
You have new mail in /var/spool/mail/root

####################################
# Configuration File for iscsi-ha #
####################################

####################################
# Declare the DRBD resource names
# in a ":" separated list like this
# resource1:resource2:resource3
####################################
DRBD_RESOURCES=iscsi1

####################################
# Set the path to the init script
# for the iSCSI target
####################################
ISCSI_TARGET_SERVICE=/etc/init.d/tgtd

####################################
# Declare the virtual "floating"
# IP address and subnet mask to be
# used for the iSCSI target
# NOTE: It is expected that the
# virtual/floating IP is on the
# same subnet as other system
# interfaces. No changes to routing
# or gateways are made by iscsi-ha
# The IP set here should work with the
# routing configuration of the environment
####################################
DRBD_VIRTUAL_IP=10.10.10.3
DRBD_VIRTUAL_MASK=255.255.255.0

###################################
# Declare the interface to bind
# the virtual IP to. Typically
# xenbr0 or xapi0, xapi1, etc..
# will vary based on environment
###################################
DRBD_INTERFACE=xapi0

##############################################
# Set iscsi-ha monitor timers
#
# MONITOR_MAX_STARTS: Threshold for when
# to assume running process is not responding.
# Sets how many failed starts to wait before
# killing any hung processes. Default = 5
#
# MONITOR_KILLALL: If MAX_MONITOR_STARTS
# threshhold is reached - set whether to kill
# processes. Default = 1
# 1 = yes, 0 = no
#
# MONITOR_DELAY: Delay in seconds between
# re-spawning iscsi-ha logic.
# Default = 10
#
# MONITOR_SCANRATE: iscsi-ha will not re-spawn
# unless all current processes are completed.
# If there are active processes while attempting
# to start a new run, iscsi-ha will wait the
# number of seconds set here before retrying.
# Each successive fail will increment a counter
# that may trigger KILLALL. Default = 10
#################################################
MONITOR_MAX_STARTS=5
MONITOR_DELAY=10
MONITOR_KILLALL=1
MONITOR_SCANRATE=5

#######################################
# Enable Logging 1=yes, 0=no
# logs written to /var/log/messages
# All log messages labeled with "iscsi-ha"
# View/Filter real time logging with:
# tail -f /var/log/messages | grep iscsi-ha
#######################################
ENABLE_LOGGING=1

##################################################################
# Email Alert Settings
##################################################################
MAIL_ON=1
MAIL_SUBJECT="HA-Lizard noSAN SYSTEM ALERT - FROM HOST: $HOSTNAME"
MAIL_FROM="root@localhost"
MAIL_TO="*******"
MAIL_SCREEN_TIME=30
#END CONFIG FILE

thanks for the assistance. i did put in ***** instead of my email address in the replies

Can you verify that xapi0 is the correct interface being used for replication?

Also, perform your test again and post log file from the Master (both iscsi-ha and ha-lizrd) so we can see what is happening.

Pulling the plug on the slave should not put it into suspeneded HA mode. This would only occur in a split pool - ie. the slave comes back up but cannot contact the master and cannot reach the hearistic IP. NOt sure why this is happening.

Also, since your master is not starting the VMs - it is likely failing to fence also. The log should show us exaclty what is happening.