Forum
Welcome, Guest
Username: Password: Remember me
  • Page:
  • 1
  • 2

TOPIC:

10.10.10.3 doesn't get activated 10 years 7 months ago #43

  • christ neeskens
  • christ neeskens's Avatar Topic Author
  • Offline
  • Posts: 18
After a power outage both servers rebooted.

Now server one is pool master and drbd master
server 2 is pool slave and drbd slave

Server 2 is in state syncTarget.

But while the sync is going on 10.10.10.3 should be working. which it isn't at the moment.
That way i could start my VM's without having to wait till the sync is completed, because this sync takes a long time (about 12 hours to sync 4TB)

I know it's a risk to have just one drbd source running, but that's an acceptable risk compared to the down-time.

Is there a possibility to
A start 10.10.10.3 while the sync is in progress?
B make it an option that it will always start 10.10.10.3 on the master if syncTarget is running, because the master should have the proper data, if not the slave will be destroyed any way.

Please Log in or Create an account to join the conversation.

10.10.10.3 doesn't get activated 10 years 7 months ago #44

DRBD syncing should not affect the promotion of the shared IP.. Can you post the output of the following?

1 - "iscsi-ha status" for both hosts

2 - "service iscsi-ha status -w" for both hosts

3 - "cat /etc/xensource/pool.conf" for both hosts

4 - on the master - capture some log output for at least one iteration of iscsi-ha - log should update roughly every 15 seconds for each iteration. "iscsi-ha log"

Please Log in or Create an account to join the conversation.

10.10.10.3 doesn't get activated 10 years 7 months ago #45

  • christ neeskens
  • christ neeskens's Avatar Topic Author
  • Offline
  • Posts: 18
1
[root@xenserver1 ~]# iscsi-ha status
-bash: iscsi-ha: command not found
[root@xenserver2 ~]# iscsi-ha status
-bash: iscsi-ha: command not found

2
[root@xenserver1 ~]# service iscsi-ha status -w
Version: IHA_1.2.12 iscsi-ha (pid 31858 31852) is running...
Version: 1.0 iscsi-ha-watchdog is stopped
[root@xenserver2 ~]# service iscsi-ha status -w
Version: IHA_1.2.12 iscsi-ha (pid 30452 30446) is running...
Version: 1.0 iscsi-ha-watchdog (pid 30511 30509) is running...

3
[root@xenserver1 ~]# cat /etc/xensource/pool.conf
master
[root@xenserver2 ~]# cat /etc/xensource/pool.conf
slave:192.168.207.200

4
iscsi-ha is no valid command.
So here is a bit of log from iscsi-cfg log on both servers, where the master indicates it can't find 10.10.10.3 which it could before the power outage

Aug 16 15:48:00 xenserver1 iscsi-ha: 26456 check_drbd_resource_state: DRBD Resource: iscsi1 in Primary mode
Aug 16 15:48:00 xenserver1 iscsi-ha: 26456 DRBD Resource: iscsi1 in SyncSource state - expected Connected state
Aug 16 15:48:00 xenserver1 iscsi-ha: 26456 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Aug 16 15:48:00 xenserver1 iscsi-ha: 26456 email: Duplicate message - not sending. Content = DRBD Resource: iscsi1 in SyncSource state - expected Connected state
Aug 16 15:48:00 xenserver1 iscsi-ha: 26456 email: Message barred for 30 minutes
Aug 16 15:48:00 xenserver1 iscsi-ha: 26456 iSCSI target: /etc/init.d/tgtd status = OK. [tgtd (pid 12679 12678) is running...]
Aug 16 15:48:00 xenserver1 iscsi-ha: 26456 local_ip_list: Local IP list returned 127.0.0.1 10.10.10.1 192.168.207.200
Aug 16 15:48:00 xenserver1 iscsi-ha-NOTICE-/etc/iscsi-ha/iscsi-ha.sh: /etc/iscsi-ha/iscsi-ha.func: line 43: -c: command not found
Aug 16 15:48:00 xenserver1 iscsi-ha: 26456 check_ip_health: 10.10.10.3 response = FAIL
Aug 16 15:48:00 xenserver1 iscsi-ha: 26456 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Aug 16 15:48:00 xenserver1 iscsi-ha: 26456 email Sending ALERT email to **REMOVED AGAINST SPAM**: check_ip_health: 10.10.10.3 response = FAIL
Aug 16 15:48:09 xenserver1 iscsi-ha: 26443 iscsi-ha already running: Attempt 1 on PIDS: 26451 26446 26445 26443
Aug 16 15:48:14 xenserver1 iscsi-ha: 26443 iscsi-ha already running: Attempt 2 on PIDS: 26451 26446 26445 26443
Aug 16 15:48:20 xenserver1 iscsi-ha: 26443 iscsi-ha already running: Attempt 3 on PIDS: 26451 26446 26445 26443
Aug 16 15:48:25 xenserver1 iscsi-ha: 26443 iscsi-ha already running: Attempt 4 on PIDS: 26451 26446 26445 26443
Aug 16 15:48:30 xenserver1 iscsi-ha: 26443 iscsi-ha already running: Attempt 5 on PIDS: 26451 26446 26445 26443
Aug 16 15:48:35 xenserver1 iscsi-ha: 26443 iscsi-ha already running: Attempt 6 on PIDS: 26451 26446 26445 26443
Aug 16 15:48:35 xenserver1 iscsi-ha: 26443 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Aug 16 15:48:35 xenserver1 iscsi-ha: 26443 email: Duplicate message - not sending. Content = iscsi-ha failed to spawn new instance after 6 attmepts. MAX_STARTS is set to 5. Check Host: xenserver1 for possible hung process

Aug 16 15:47:23 xenserver2 iscsi-ha: 30511 iscsi-ha Watchdog: iscsi-ha running - OK
Aug 16 15:47:24 xenserver2 iscsi-ha: 20451 Spawning new instance of iscsi-ha
Aug 16 15:47:24 xenserver2 iscsi-ha: 20620 Checking if this host is a Pool Master or Slave
Aug 16 15:47:24 xenserver2 iscsi-ha: 20620 This host's pool status = slave:192.168.207.200
Aug 16 15:47:24 xenserver2 iscsi-ha: 20616 auto_plug_pbd: Found LVMoISCSI SR List: 2e56fc62-c9d7-28f8-c1aa-da6734138f16
Aug 16 15:47:24 xenserver2 iscsi-ha: 20620 local_ip_list: Local IP list returned 127.0.0.1 192.168.207.201 10.10.10.2
Aug 16 15:47:24 xenserver2 iscsi-ha: 20620 iSCSI target: /etc/init.d/tgtd status stopped. Expected Stopped . [tgtd is stopped]
Aug 16 15:47:24 xenserver2 iscsi-ha: 20620 DRBD Running on this host: version: 8.3.15 (api:88/proto:86-97) GIT-hash: 0ce4d235fc02b5c53c1c52c53433d11a694eab8c build by root@XS2, 2013-08-04 23:01:14 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r
ns:0 nr:1591181056 dw:1591172864 dr:0 al:0 bm:97073 lo:129 pe:14564 ua:128 ap:0 ep:1 wo:b oos:353892536 [===============>....] sync'ed: 81.9% (345596/1899476)M finish: 1:26:29 speed: 68,176 (79,092) want: 102,400 K/sec
Aug 16 15:47:24 xenserver2 iscsi-ha: 20620 check_drbd_resource_state: DRBD Resource: iscsi1 in Secondary mode
Aug 16 15:47:24 xenserver2 iscsi-ha: 20620 DRBD Resource: iscsi1 in SyncTarget state - expected Connected state
Aug 16 15:47:24 xenserver2 iscsi-ha: 20620 Mail Spool Directory Found /dev/shm/iscsi-ha-mail
Aug 16 15:47:24 xenserver2 iscsi-ha: 20620 email: Duplicate message - not sending. Content = DRBD Resource: iscsi1 in SyncTarget state - expected Connected state
Aug 16 15:47:24 xenserver2 iscsi-ha: 20620 email: Message barred for 30 minutes

Please Log in or Create an account to join the conversation.

Last edit: by christ neeskens.

10.10.10.3 doesn't get activated 10 years 7 months ago #46

  • christ neeskens
  • christ neeskens's Avatar Topic Author
  • Offline
  • Posts: 18
Salvatore,

Thanks for the phone call!
Should I replace the line we changed back to it's original?

`$PING -c $2 $1`
back to
$PING -c $2 $1 1> /dev/null

Please Log in or Create an account to join the conversation.

10.10.10.3 doesn't get activated 10 years 7 months ago #47

YEs - change to the original for now. We will duplicate the scenario and post an update here.

Please Log in or Create an account to join the conversation.

10.10.10.3 doesn't get activated 10 years 7 months ago #48

  • christ neeskens
  • christ neeskens's Avatar Topic Author
  • Offline
  • Posts: 18
Thanks.

I also found a small typo in the log mechanisme.

Aug 16 17:11:20 xenserver1 iscsi-ha: 6543 email: Duplicate message - not sending. Content = iscsi-ha failed to spawn new instance after 6 attmepts. MAX_STARTS is set to 5. Check Host: xenserver1 for possible hung process

attmepts should be attempts

Please Log in or Create an account to join the conversation.

  • Page:
  • 1
  • 2