TOPIC:

Hello,

i had a problem with the last xenserver 9 update only partially committed on my secondary so i had to make a new installation.

As i cant get the secondary into the pool as the storage backend is different i cant use the install script.

Therefore i would like to ask how to approach a situation where you need to replace on of the 2 servers in a ha-lizard system.

In general i think one has to:

- install drbd
- install HA-ISCSI first, restore the iscsi, tgtd (?) and drbd config
(oops, i forgot to backup the tgtd conf but i guess i can reconstruct this one from the masters version)
- Then get drbd online as secondary
- Then "somehow" install ha-lizard manually
I did see that the pool server IDs are stored under /etc/halizard.
The secondaries UUID did change so where do i have to change it?
Only in "/etc/ha-lizard/state/host.someUUId.vmlist.uuid_array"? Could i just rename it to the new servers UUID?

Is it safe to unplug the HA-Bond while re-installing drbd and iscsi?

We would also consider sponsering/donating to a revovery script if you care to write one.

cheers,

Robin

Hi Robin, the most important part of the procedure would be syncing your data from your live master. The rest of it is pretty straight forward.

Regarding your question about UUID change - this should not matter as we do not rely on storing this data other than for HA recovery. Since your UUID changed, you will end up with some harmless stale data in the /state directory. It is safe to delete the entire contents of this directory as new updated data will written to it within a few seconds of clearing its contents.

Below are detailed instructions - but, you should be able to simply run the installer again on the new slave. The only piece that will not work is syncing data from your master to your slave. You can manually start the initial DRBD sync form the master to the slave when ready.

Have you tried joining the secondary server to the xenserver pool? The primary should be exposing the iscsi target just like any SAN and the secondary should be able to connect.

Here are the basic steps if you attempt to build manually:

We have the steps documented in our reference design available on our web site.. You can refer to that for more detail. the general steps are below (assuming you are running XenServer 6.5 - if you are running 6.2 or older, refer to the reference design for detailed instructions for installing DRBD correctly).

- join slave to the pool

- copy SW from your Master to your Slave
scp -r /etc/ha-lizard root@your.slave.host:/tmp
scp -r /etc/iscsi-ha root@your.slave.host:/tmp

- install ha-lizard packages
/tmp/ha-lizard/scripts/install --nostart
/tmp/iscsi-ha/scripts/install --nostart

- install tgt
yum --enablerepo=base install scsi-target-utils -y

- if you are on xenserver 6.5 - install drbd utils from the included RPMs in /tmp/iscsi-ha/RPM/"your kernel version"

- if you are on xenserver 6.2 or older - refer to the reference design

- copy your /etc/lvm/lvm.conf from your master to your slave
- copy your etc/drbd.cond from your master to your slave
- copy your tgt configutaion from your master to your slave
- clear the lvm cache and then vgscan

- connect DRBD resource
- from master - sync data from master overwriting the peer
- start iscsi-ha and ha-lizard services.

// Deleted, see below

Well, live could had been a lot more easy by doing a

after installing the secondary!

I couldnt connect to the master because the masters xen network configuration is bridged mode for ha-lizard but the secondary server had been still in the default xen-openswitch mode. Therefore the secondary couldnt join pool.

I accidently thought it would be because iscsi wouldnt had been installed on the secondary.... *sic*

Well, the secondary is in the pool now, both have their iscsi storage back thanks to a "iscsi-cfg become-primary" on the master.

So now i can re-install ha-lizard on the secondary. I did comment out the drbd storage initialisation in the installer-script. That should be enough if i manually sync with the drbd master node later, right?

We are still in on sponsoring/paying for an recovery script.

Thanks for posting an update.

Since you are introducing a new disk to DRBD, syncing should start automatically after creating metadata on the secondary and then attaching the secondary. Let us know if your issue is now totally resolved.

Regarding a recovery script - most of the logic needed is already in the installer. It could be adapted with some tweaking, however, I am not keen on automating DRBD operations on a live system.. that is better left as a manual job to avoid the possibility of destroying someones data.

In theory - once your secondary was rejoined to the pool, the installer script would have worked perfectly fine on the secondary since it had no data to begin with.