Forum
Welcome, Guest
Username: Password: Remember me
This is the optional category header for the Suggestion Box.

TOPIC:

Xenserver crashed and halizard seems broken 6 years 1 month ago #1533

  • engine411
  • engine411's Avatar Topic Author
  • Offline
  • Posts: 11
I have a 2-host pool with xenserver 7.2 and HAL. The power to my master in the pool got cut and now everything is down. There was no graceful failover that HAL is supposed to make happen.

Currently, my pool is broken and the two xenserver can’t connect to xencenter. I have physical access to the servers and the consoles show empty management interfaces, so I did the emergency network reset but the servers don’t show the settings I assigned during the reset.

When I run “watch cat /proc/drbd “ on the top server, it returns no process found or “no such file or directory” or something like that.

Do I have options besides wiping the two xenservers and starting over with a new install?
The topic has been locked.

Xenserver crashed and halizard seems broken 6 years 1 month ago #1535

  • Salvatore Costantino
  • Salvatore Costantino's Avatar
  • Offline
  • Posts: 722
If you can get network connectivity to the management IPs, then it should be recoverable.

Did you happen to lose power to the network switch at the same time? If the slave was unable to reach your configured heuristic IP then it would not takeover any services. This is intentional and meant to prevent split brain and would result in your cluster being down.

Regarding /proc/drbd missing. That simply means that DRBD is not running. The file will appear after DRBD has started.
The topic has been locked.

Xenserver crashed and halizard seems broken 6 years 1 month ago #1536

  • engine411
  • engine411's Avatar Topic Author
  • Offline
  • Posts: 11
No, the switch was up the whole time. One server is the only device that lost power.

Any idea why the xenserver management IPs are empty in the console?
The topic has been locked.

Xenserver crashed and halizard seems broken 6 years 1 month ago #1537

  • engine411
  • engine411's Avatar Topic Author
  • Offline
  • Posts: 11
Ok I’m able to get xencenter connecting to the master and slave now after doing some google-fu and some local console commands to force stop HA and force the master to take the master role again.

Now I can’t get the storage to get plugged in. Are there commands I can run to get the HAL started and repairing itself?
The topic has been locked.

Xenserver crashed and halizard seems broken 6 years 1 month ago #1538

  • engine411
  • engine411's Avatar Topic Author
  • Offline
  • Posts: 11
I can’t repair the iscsi target in xenserver and when I try to add a new iscsi storage, it can’t connect to the target ip.

Looking at the storage itself, xencenter says the SR has no PBD’s.

Any help?
Last edit: by engine411.
The topic has been locked.

Xenserver crashed and halizard seems broken 6 years 1 month ago #1539

  • Salvatore Costantino
  • Salvatore Costantino's Avatar
  • Offline
  • Posts: 722
Did you happen to have the XenServer HA enabled in your pool? If so, disable it and keep it disabled.

You can try the following to try to restart services on the master:

service iscsi-ha-watchdog stop
service iscsi-ha stop
service ha-lizard-watchdog stop
service ha-lizard stop
service drbd stop
service tgtd forecedstop

if you manage to get all of services stopped then try this which will bring everything back up in the correct order:

service iscsi-ha-watchdog start

Then see if the master can connect to the iscsi storage (the slave may still be broken at this point, but this will get you most of the way to getting your VMs up). I don't recommend adding a new SR. The originally defined iscsi SR should be OK
The following user(s) said Thank You: engine411
The topic has been locked.