TOPIC:

We have not seen this issue at all in our development environment.

Would you mind sharing your setup details so that we can try to replicate the issue?

- Did you use our automated installer to build the system or did you build by hand? If installer was used - what version number?

- was the installer run more than one time? Check the bottom of the file /etc/tgt/targets.conf if the target is defined more than one time.

If that checks out OK - can you send the management IPs used in your setup and replication IPs (complete with subnet information).. Also the XenServer version and any patches applied. we can then build a system exactly like yours to see if we can reproduce it.

Joe,
We have been able to reproduce this issue. It seems to occur under the following conditions:

- when installing XenServer, you select more that one disk partition for VM storage
- when running the halizard noSAN installer - you answer YES to convert the local storage to halizard managed storage.

Can you confirm that the above is true for the installation you had trouble with?

The core issue is this: Our installer is expecting a single block device for the storage. In the above scenario, XenServer has created an LVM Volume Group with 2 or more physical volumes because more than one disk partition was selected for VM storage. Simply put, the installer is not written to deal with this scenario and cannot properly deal with the 2+ disk partitions leftover after the backing Volume Group is removed.

We will look into dealing with this scenario in an automated way. In the meantime, you have a couple of options:

- try to reinstall - this time selecting only a single partion for VM storage when prompted by the XenSever installer

- before running our installer - manually remove the local storage from XenServer. You will end up with 2 or more available block devices. Use them to create a Volume Group and then pass that VG as the path to the local storage when our intaller prompts for it.

Hi,

I'm having this issue right now, with 2 just installed an re-installed OVH servers in a vrack, running XenServer 6.5.

I'm using halizard.org/release/noSAN-combined/hali...osan_installer_1.4.7

I think the problem is about to unplug or detach the local storage, that is /dev/md3.

This is what I get:

---

| Convert local storage to iSCSI-HA storage? <yes or no> |

---

yes
Hostname detected = HA-Liz01
Local Storage SR UUID discovered = eaf84c4a-8e64-b2d4-9649-ad1b10c15835
Local Storage PBD UUID discovered = 9f9b629f-cc65-4275-c08f-6dc7245ccdb1

Press Enter to Continue


Unplugging PBD 9f9b629f-cc65-4275-c08f-6dc7245ccdb1
There was an SR backend failure.
status: non-zero exit
stdout:
stderr: Traceback (most recent call last):
File "/opt/xensource/sm/LVMSR", line 2130, in ?
SRCommand.run(LVHDSR, DRIVER_INFO)
File "/opt/xensource/sm/SRCommand.py", line 345, in run
ret = cmd.run(sr)
File "/opt/xensource/sm/SRCommand.py", line 110, in run
return self._run_locked(sr)
File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
rv = self._run(sr, target)
File "/opt/xensource/sm/SRCommand.py", line 325, in _run
return sr.detach(self.params)
File "/opt/xensource/sm/LVMSR", line 668, in detach
raise Exception("SR detach failed, please refer to the log " \
Exception: SR detach failed, please refer to the log for details.

error detected while unplugging PBD 9f9b629f-cc65-4275-c08f-6dc7245ccdb1. Exiting

---

| Error converting local SR to usable block device |
| Enter full path to backing block device to be |
| used for HA storage and press <Enter> (ex. /dev/sdb) |

---

/dev/md3
Backing block device set to /dev/md3

---

| Enter LUN to be used for exposing iSCSI storage and press <Enter> |
| (Leave blank to accept default value of 10) |

---

LUN set to 10

Backing up LVM configuration file /etc/lvm/lvm.conf to /etc/lvm/lvm.conf.halizard_restore
Reading all physical volumes. This may take a while...
No volume groups found

---

| Enter heuristic IP address used by HA-Lizard. |
| This can be any IP address reachable by this host |
| by traversing the XenServer management network and press <Enter> |

---

149.202.90.254
Heuristic IP set to 149.202.90.254

Press <enter> to continue installation

Successfully updated FENCE_ENABLED to value: 1
Successfully updated FENCE_HEURISTICS_IPS to value: 149.202.90.254
Successfully updated FENCE_MIN_HOSTS to value: 2
Successfully updated FENCE_QUORUM_REQUIRED to value: 1
Successfully updated FENCE_USE_IP_HEURISTICS to value: 1
Successfully updated MONITOR_DELAY to value: 15
Successfully updated MONITOR_MAX_STARTS to value: 20
Successfully updated XAPI_COUNT to value: 2
Successfully updated XAPI_DELAY to value: 10

---

| Initializing DRBD.. This host will wait until its peer connects. |
| Installation will resume after the DRBD peers connect |

---

1+0 records in
1+0 records out
1048576 bytes (1,0 MB) copied, 0,00112544 seconds, 932 MB/s
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
Starting DRBD resources: [
create res: iscsi1
prepare disk: iscsi1
adjust disk: iscsi1:failed(attach:10)
adjust net: iscsi1
]
..........
***************************************************************
DRBD's startup script waits for the peer node(s) to appear.
- In case this node was already a degraded cluster before the
reboot the timeout is 0 seconds. [degr-wfc-timeout]
- If the peer was available before the reboot the timeout will
expire after 0 seconds. [wfc-timeout]
(These values are for resource 'iscsi1'; 0 sec -> wait forever)
To abort waiting enter 'yes' [ 257]:
.
Synchronizing storage with peer/slave host1: State change failed: (-2) Need access to UpToDate data
Command 'drbdsetup primary 1 --overwrite-data-of-peer' terminated with exit code 17
Starting ha-lizard: [ OK ]
Starting ha-lizard-watchdog: [ OK ]

Starting iscsi-ha: [ OK ]
Starting iscsi-ha-watchdog: [ OK ]

---

| The final step is to create a new storage repository of type iSCSI from XenCenter |
| targeted to 10.10.10.3 to complete the noSAN installation. |

---

Then, when I try to create the iSCSI SR for the Pool in XenCenter I get the "Login failed" message.

Local storage is no in use, the installation is completely blank, and all available updates for XenServer are applied.


Thanks in advance,

The initial error when the script tries to unplug the PBD triggers the remaining errors. It's likely that the device specified (/dev/md3) after the failure to remove the local SR is still mounted which would cause the DRBD setup to ultimately fail.

I would suggest manually removing the local SR after a clean XS install to avoid the error and then specify the block device manually (making sure it is not mounted or used by anything else).

This will be a rather long post, I thought it might be useful to detail my installation experience as it relates to this problem.

I encountered two problems during installation, in this order:
1. Connected to iSCSI target but LUN could not be found
2. Could not connect to iSCSI target, with XenCenter reporting 'login failed, verify CHAP credentials'


Problem 1

After installation, I could connect to the iSCSI target but could not find any LUNs.

Cause

An error had occurred during installation due to the fact that I had specified a device for the storage that was already mounted in XenCenter as a storage repository. For reference, I had specified a single drive at '/dev/sdb'; this is a 1TB HDD with hardware level RAID 1.

Solution

I noticed in the installation read-out that HA-Lizard reported a problem unplugging the device. I concluded that this was due to the fact that it was already mounted as an SR and so I manually unplugged the drives, deleted the PBDs, and destroyed the Storage Repositories. When I re-ran the installation, it completed without any errors being reported.


Problem 2

After installation, I could not connect to the iSCSI target. XenCenter reported 'Login failed, verify CHAP credentials'.

Cause

My problem appears to have come from the fact that I had run the installation twice; with HA-Lizard attempting to create the same iSCSI target multiple times. Why this manifested as an authentication error message... I have no idea.

Solution

First, I restarted the servers as some users have reported problems with iSCSI targets locking up. This did not resolve the problem but I mention it as it may or may not be important in recreating the scenario.

Then I inspected the iSCSI target specification at /etc/tgt/targets.conf; I found that the HA-Lizard configuration had been appended to the configuration each time the installation was run. I cleared the duplicate entries and restarted the service.

/etc/init.d/tgtd restart

I performed this on both hosts. The master host restarted the service without error whilst the slave host reported an exit code of 22 when attempting to start tgtd. After looking into it a bit further my best guess is that this is expected behaviour since the master host is active and will therefore be the authoritative source for the iscsi target. Once the storage has finished syncing I'll try killing the master host and checking to see if the tgtd service starts on the slave.

Thanks for posting your detailed test results.

To clarify your last point about failing to start TGT on the slave. The iSCSI target should only run on a single host. Even if you managed to get it started on the slave, the iSCSI-HA service would detect it, stop it and send an alert.