TOPIC:

First of all, I would like to thank you for the great products (HA-Lizard and ISCSI-HA). I really appreciate.

Currently, I am evaluating. Setting up and installing by the given no_san script on 2 identical hosts as following:

Server 1:
Role >>> Master pool
TGTD >> Running (as expected)
DRBD >> Primary
Virtual IP (VIP) for storage >> located here (as expected)
IP_HEURISTICS is pingable
And 1 VM running (CentOS 6.7)

Server 2:
Role >> Slave
TGTD >> stopped (as expected)
DRBD >> Secondary
Virtual IP (VIP) for storage >> located at Server1 (as expected)
IP_HEURISTICS is pingable
No VM running

I leaved every configuration as default. Every scenario I have tested work perfectly (e.g. forcefully shutdown servers and unplug Management LAN cable of POOL SLAVE). However, just one scenario that I think it might be incorrect behavior which is unplugging Management LAN cable of MASTER POOL.

As soon as I unplugged Management LAN cable of MASTER POOL (in this case, server 1). The following activates took place on Server 2 :

1. switched to Master POOL
2. VM started on Server2
3. TGTD still stopped ( expected to be running here)
4. DRBD still be secondary
5. VIP of storage still be at Server 1 (expected to be running here)

While management LAN cable still unplugged, I have also verified the status of server 1 by directly access to Xenserver Console. The following occurred:

1. Still be Master POOL
2. VM still running
3. TGTD still running
4. DRBD still be primary
5. VIP still be here

Note that I can access VM and I can read/write files on that VM as well. So would you please clarify my following questions?

1. Is that expected behavior of HA-Lizard/iSCSI-HA? However, I believe that it is not. It seems I am encountering spilt brain situation as there are 2 active Master POOLs with identical VM running on them. Server 2 might have not fenced Server1 correctly before it took over to MASTER POOL.

2. If it is not the expected behavior, please explain the correct behavior for this scenario. (I have read through your manual, it mentioned that POOL Salve should eject the failed Master POOL off before it promotes itself to MASTER. But I don’t think it correctly happens here in my lab test)

3. Can you provide me some suggestion where I should further focus more on the troubleshooting.

I will provide the all log file and configuration as soon as I can access the servers.

Thank you in advance for your help.

Thanks for sharing your test results.

Can you try your test again, but this time first set fence_reboot_lone_host=1 with the following command
"ha-cfg set fence_reboot_lone_host 1"

This should force the Master to remove itself from the pool which would then allow the slave to promote the DRBD resource and start exposing the storage to the VM now running on the slave.

*** update to previous answer ***
fence_reboot_lone_host=1 will not work in a 2 node pool when using the current release of ha-lizard.

This particular test case really should be using HW fencing to ensure that the Master is not rejoined to the pool.

Pulling the management link becomes an issue IF the management link is restored after the slave has already promoted itself to master. In this scenario you end up with 2 masters. To make matters worse, since the original master never really failed, it held the primary position of the storage which prevents the secondary (new master) from taking on the storage primary role. The only predictable solution would be to power down the master via an out of band connection (like ILO, DRAC, power strip, etc...). With STONITH properly configured, the slave would never take on the storage role or start any VMs unless it has first successfully powered down the original master.

Thank you so much for your response. I will follow your recommendation.

However, Is it possible if I would like to run ha-lizard with the version that fence_reboot_lone_host still working? Just in case HW fencing is not possible for me.

We may have a release as early as this week. The download link will be posted here once ready

Thank you so much. Waiting for new release.