TOPIC:

attached are the logs from an orderly shutdown of the slave and then one when i pull the plug on the slave

Your quorum IP is not responding so the remaining host does not have quorum and will not start the VMs. see snippet from log below.

Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 check_quorum: Heuristic IP: 192.168.1.253 Response = Fail
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 Total enpoints checked = 1 with total successful replies = 0
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 check_quorum: Failed to add additional vote by checking heuristics
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 check_quorum: Minimum number of hosts needed to allow fencing = 1 + 1
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 check_quorum: 1 Hosts found. Minimum needed 1 + 1. Fencing blocked
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 fence_host: Remaining hosts do not have quorum - setting FENCE_BLOCK
Jul 23 11:15:36 HA-xen001 ha-lizard: 12967 fence_host: Fencing is disabled or blocked. Check configuration or pool state. FENCE_ENABLED = 1, FENCE_BLOCK = 1, NUM_HOSTS = 2, FENCE_MIN_HOSTS = 2 - cannot fence host 3b3355ca-54ba-48ff-85a5-1c9724b39c15
Jul 23 11:15:49 HA-xen001 ha-lizard: 12967 check_slave_status: fence_host return status = 1
Jul 23 11:15:49 HA-xen001 ha-lizard: 12967 check_slave_status: fence_host failed to fence failed host: 3b3355ca-54ba-48ff-85a5-1c9724b39c15
Jul 23 11:15:49 HA-xen001 ha-lizard: 12967 check_slave_status: Host: 3b3355ca-54ba-48ff-85a5-1c9724b39c15 power state unknown
Jul 23 11:15:49 HA-xen001 ha-lizard: 12967 Function check_slave_status failed to fence failed host.. checking whether to attempt starting failed VMs
Jul 23 11:15:49 HA-xen001 ha-lizard: 12967 FENCE_HA_ONFAIL is set to: 0
Jul 23 11:15:49 HA-xen001 ha-lizard: 12967 FENCE_HA_ONFAIL is set to: 0, not attempting to start VMs

what should the quorum ip be set to the virtual ip of the iscsi backend 10.10.10.3 ?

the quorum ip was set to 192.168.1.253 per the instructions on pg 18 of the how to i set it back to the default of 8.8.8.8 and now the vm's migrate over to the master durning a hard shutdown. the vm's do reboot. is it possible for them to migrate over without a reboot and continue to run as if there were no interuption

Nice to see you got it working.
In the event of a crash (as in your test scenario) the VM will start on the surviving node. It will be a cold start since the VM that was running crashed.

Regarding the heuristic IP address. You should read the HA-Lizard documentation regarding this topic and avoid using 8.8.8.8 in a production system.

ok thanks for clarifying the vm restart. I will check the documentation in regards to the 8.8.8.8