Citrix: Export to network drive really slow

For one of our customers I was asked to troubleshoot an application that exports data to several file types, including Microsoft Excel’s XLS file type.

Environment

The client runs Citrix XenApp 5 on Windows Server 2003 (x86). All user homedrives are mapped to the H-drive.

Problem

The application lets you perform queries on your Oracle database, analyze and format the results and prepare those for presentation. When the application exports an XLS-file to the H-drive, it freezes for 5 minutes and then responds normally again. However, when the same application exports a TXT- or CSV-file to the same drive, it freezes for just a few seconds.

Troubleshooting

Since the application is a ThinApp, I first tried to make the ThinApp smaller. Unfortunately, even after resizing it to half of the original size, this didn’t solve the problem. I went back to desktop environment and, while running the export, I ran Process Monitor. I filtered out the applications process and saw several successful entries where the application writes the file to the H-drive. However, most entries only had a length of 2 bytes.

I switched over to my virtual machine and did the same thing there. Process Monitor showed me that the length was 4096 bytes.

I also noticed that no I/O Flags were set in the results I got from the desktop environment.

Thankfully, Google is your friend, so I searched for the missing I/O Flags. The first result read “Reading file over network slow due to extra reads” which closely resembled the problem I was faced with. According to the top answer, the problem was most likely caused by so called opportunistic locks (or oplocks).

Opportunistic Locks

In short, oplocks are requests from the client to the server. From the point of the client, they are opportunistic. More on oplocks can be found at the following websites:
http://msdn.microsoft.com/en-us/library/aa365433%28VS.85%29.aspx
http://msdn.microsoft.com/en-us/library/cc308442.aspx

My next search was for opportunistic locks in combination with Windows Server 2003. It resulted in a Microsoft Knowledge Base article: http://support.microsoft.com/kb/296264/en-us. The article refers to a DWORD-value in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MRXSmb\Parameters\ called “OplocksDisabled”. Just to be clear: if set to 1, the oplocks are disabled and if set to 0, the oplocks are enabled (which is the default setting). I looked up the DWORD-value in the registry of the desktop-environment.

The option is disabled. I then looked up the value on my VM.

The value wasn’t even there, which means the default setting is being used. To see if the problem would occur on my VM, I made manually added the DWORD-value and set it to 1. When that is done, the change requires the system to reboot. So reboot, start the application, run the query and export it to the H-drive. As expected, the application stopped responding. I opened up the Task Manager and the memory was slowly rising. I killed the process, as this pretty much confirmed that this was the solution.

Just to be absolutely sure, I logged onto to a test server with my administrator account. I changed the DWORD-value, which, on these servers, by default is set to 1. I made sure the server wouldn’t be picked up by the Provisioning Server after the required reboot and rebooted the test server. When it was back up, I redirected my session to the test server and logged onto the desktop environment with my regular account. Started the ThinApp, ran the query and clicked Finish in the Export window. The result was positive; within 5 to 10 seconds the XLS-file was generated.

Citrix Best Practice

There are several websites out there, e.g. this one, pointing out best practices to use in a Citrix Xenapp-environment. Setting this registry value is one of them. This problem proves that, before setting a value at all, you should first test if the value works for your environment.