Forum
Welcome, Guest
Username: Password: Remember me

TOPIC:

iSCSI SR Broken if I reboot Master Server 7 years 5 months ago #994

Just catching this old thread as we have a similar problem after maintenance on our master. Following the "iscsi-ha_1.5_final" guide the iscsi virtual storage in XenCenter stays broken. We are still in manual mode and the masters storage is still secondary.

The master could not ping see the floating ip 10.10.10.3.

How could I find out what is wrong.

If I change the slaves storage role to secondary and the masters role in primary the storage link comes back, and I could ping the floating ip???

Any hint would be appreciated.

BR Andreas

Please Log in or Create an account to join the conversation.

Last edit: by ajmind. Reason: changed subject

iSCSI SR Broken if I reboot Master Server 7 years 5 months ago #995

  • Salvatore Costantino
  • Salvatore Costantino's Avatar
  • Offline
  • Posts: 722
This is likely related to xenserver networking. Try restarting the toolstack on the master and then return storage by exiting manual mode.

Please Log in or Create an account to join the conversation.

iSCSI SR Broken if I reboot Slave Server 4 years 1 month ago #1972

We have faced a different issue today. Our master server was rebooted automatically, and come up with the following message,

"a storage repository could not be attached when server starts"

All the VM's hence failed to start automatically on the Slave. But starting manually works and we were able to bring all VM's up on Slave. Master server still seems to have the same SR issue.

Can ping 10.10.10.3 from both hosts and telnet to the SR IP's. Can you advice us here.

```````````````
| iSCSI-HA Version IHA_2.2.0 |
| Mon Feb 3 01:58:19 EST 2020 |

| iSCSI-HA Status: Running 5276 |
| Last Updated: Mon Feb 3 01:58:19 EST 2020 |
| HOST ROLE: MASTER |
| DRBD ROLE: iscsi1=Primary |
| DRBD CONNECTION: iscsi1 in Connected state |
| ISCSI TARGET: Running [expected running] |
| VIRTUAL IP: 10.10.10.3 is local |
Control + C to exit


| DRBD Status |

| version: 8.4.10 (api:1/proto:86-101) |
| srcversion: CE41F2FAB70AE3534BF40A6 |
| 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r
|
| ns:12440760 nr:0 dw:7390904 dr:52103644 al:5208 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 |




| iSCSI-HA Version IHA_2.2.0 |
| Mon Feb 3 02:50:11 EST 2020 |

| iSCSI-HA Status: Running 30402 |
| Last Updated: Mon Feb 3 02:50:06 EST 2020 |
| HOST ROLE: SLAVE |
| VIRTUAL IP: 10.10.10.3 is not local |
| ISCSI TARGET: Stopped [expected stopped] |
| DRBD ROLE: iscsi1=Secondary |
| DRBD CONNECTION: iscsi1 in Connected state |
Control + C to exit


| DRBD Status |

| version: 8.4.10 (api:1/proto:86-101) |
| srcversion: CE41F2FAB70AE3534BF40A6 |
| 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r
|
| ns:0 nr:1027637380 dw:1027637380 dr:0 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 |
```````````````

Please Log in or Create an account to join the conversation.

iSCSI SR Broken if I reboot Slave Server 4 years 1 month ago #1974

  • Salvatore Costantino
  • Salvatore Costantino's Avatar
  • Offline
  • Posts: 722
can you check to ensure that HA is enabled? The slave will not start VMs unless certain conditions are met.
ha-cfg status

1- HA is enabled (not XCP/XenServer HA, but HA-Lizard HA)
2- master must be unreachable
3- slave must see the configured heuristic IP address
4- slave must have HA enabled AND must promote itself to the new master

Depending on your settings, and if your master reboots very quickly, it is also possible the master reappeared before the slave finished its validations that must all pass in order for the slave to promote itself to master.

Lastly, ha-lizard/iscsi-ha is designed to recover from HW failures. A graceful shutdown of the master introduces delays in the slave's ability to recover services.

Another operational characteristic that should be considered is that, by default, only the master is responsible for starting VMs. If the slave remained a slave (did not promote), it would not start VMs automatically. This behavior can be changed. See snippet from documentation below.

slave_vm_stat
By default, only the pool master will check the status of all VMs managed by
this script and attempt to start a VM that is not in the running state. Setting
SLAVE_VM_STAT to 1 will cause any pool slaves to also check all VM statuses and
attempt to start any VM not in the running state. Default = 0 In a large pool
many hosts may attempt to start the same VM the first host to attempt will
succeed, others will be safely declined. Enabling may create many unnecessary
duplicate processes in the pool.

Please Log in or Create an account to join the conversation.

iSCSI SR Broken if I reboot Slave Server 4 years 1 month ago #1976

Hi Salvatore,

It seems the command ha-cfg is no longer available after the latest update. Is there any other commands available to check if HA is enabled?

In the present situation, Even though both Hosts are up, master server seems as not connecting to the Pool within Xencenter. Also this error was in Xencenter logs, "a storage repository could not be attached when server starts".

Can you please advice us on how to overcome this and add master server successfully to the pool?
Attachments:

Please Log in or Create an account to join the conversation.

iSCSI SR Broken if I reboot Slave Server 4 years 1 month ago #1977

  • Salvatore Costantino
  • Salvatore Costantino's Avatar
  • Offline
  • Posts: 722
It's starting to look like ha-lizard is not installed.. And possibly iscsi-ha too, which would be responsible for exposing the storage to the network.

You mentioned that this occurred after an update. Was it an XCP update? If so, which version did you upgrade from and to.. If a major version upgrade was performed (say 7 -> 8), you would need to reinstall ha-lizard and iscsi-ha.

Please Log in or Create an account to join the conversation.