TOPIC:

Hi

We are evaluating Xen with HA-lizard to replace our existing install setup which consists of two Ubuntu 12.04 servers with DRBD syncing across the two servers. We run Virtual Box in headless mode and run a Windows server as a VM. This setup has been working quite well. The reason we are looking to move to Xen is it is a simpler installation, the Xen management tools are better and the performance could be better.

Our evaluation setup is: Two Dell R430 servers. Each server has a 2 disk raid 1 partition where Xen is installed and a 3 disk raid 5 partition set up as a iSCSI /DRBD shared disk, 1 global hot spare disk.The clustering and HA works well so far all our tests pass except one.

There is an edge test case which is not passing.
The test is as follows:
Create a text file add some text, save and close the file.
Immediately re-open the text file and confirm the text is present in the file.
Within 2 seconds of opening the file pull the power from the server.
The expected behavior is that the vm starts on the other host server and that the text file contains the text we added.
What we are seeing is the vm starting on the other server but the text is not in the file.
In reality this scenario is unlikely to happen as we have a ups attached to the server which will gracefully shut the servers down if mains power fails.

Our Ubuntu set up had the same problem. It turned out that Virtual Box has a 5M cache before it flushes to disk. We got around this by creating a samba share on the host Ubuntu system and then making that available to the Windows server. We were able to pass the edge test with that setup.

My question is, is there a better way to do this using Xen Server and HA-Lizard?

Cheers

Daryl

It's hard to tell where your data is being cached.
If your test VM is a windows machine, have you checked to ensure disk write cache is disabled?

Or

If you test VM is a linux machine try performing a "sync" from the shell right before pulling the power.

Regardless, if I get a moment tomorrow, I will test your scenario with a linux guest.

Hi

The VM is Windows 2012 R2. It will be running SQL Server 2012. We are not writing a text file through a program, we are just using notepad.
We are doing this test because we are concerned about SQL server not flushing to disk before the server looses power.

Is the disk write cache is setting on the server or OS?
We have a RAID card in the server with on board cache and battery back up.


Thanks for your reply.

Cheers

Windows does have a disk write caching feature.. You should check to ensure that it is disabled. (See attached image - setting can be found in device manager)

Also - if you have not installed the windows PV drivers, you may want to try that also as QEMU may be caching.

As a last resort - disable BBC on your raid controller to see if that is causing it.

Hi

So we disabled the write cache in Windows Server 2012 and made sure the appropriate XenServer Tools package was installed. But were still experiencing data loss if the power is pulled out within 2 seconds of the text file being written. Then we disabled the Battery Backed up cache on the server with write through enabled and still experienced the same problem.

At this stage the thought process was that the problem was either with

1. Xen Server Caching
2. Windows Server 2012

To diagnose further we did some testing on a single Xen Server node with the following operating systems

1. Windows Server 2012 R2 64-bit Standard
2. Windows Server 2003 32-bit Enterprise
3. Ubuntu Serevr 64-bit 12.04 LTS

With all three operating systems a text file was created and saved. Then some text was written into the file with a time stamp and as soon as it was saved the power on the server was pulled within 2 seconds.

The results were as follows

1. Windows Server 2012 R2 64-bit Standard: The file never had the text with the time stamp in it
2. Windows Server 2003 32-bit Enterprise: The file always had the text with the time stamp
3. Ubuntu Serevr 64-bit 12.04 LTS: The file always had the text with the time stamp

The only time when the text and the text and time stamp did not show up in Windows 2003 and Ubuntu was when the save button was pressed at the same time as the power was pulled. I guess this scenario is also prone to the order of events; i.e. was the save button pressed first or if the power was pulled first.

With further investigation it appears that the issue is with Windows Server 2012 where there is some sort of a cache that requires some time before it is flushed to disk. See KB2855336 and KB2853952. Installing all the windows updates specifically the ones listed resulted in th cache flush time being reduced to 5 seconds but this issue still persists.

So the question is how can we configure Windows 2012 to not use its internal cache and flush to the disk straight away providing us with similar results to Windows 2003 and Ubuntu.

Another train of though was to use a shared storage device or a network attached storage which might provide writh through access to the disk and not be prone to this caching issue we are experiencing.

Has anyone experienced this before? And does anyone know how to disable all caching in Windows Server 2012 apart from what has already been done?

Regards,

Thanks for the detailed test results. You should consider posting your question on the Citrix forum as well as there are surely many Windows Server users there.