Imprivata’s Blue Screen of Death?

A while back we had strange incident occurring on a couple hundred of Windows Embedded systems build into Computer on Wheels (COWS). In this case the machine would boot up and where it normally should show the Imprivata login screen it now didn’t show anything at all!

When taking a closer look we discovered that the Imprivata agent failed to launch. And as the local explorer shell was disabled the machine now just showed a blank screen.

After enabling the explorer shell and rebooting we could do a more detailed analyses of what was happening.

 

 

There we noticed that the agent was grayed out with a yellow triangle in it.

When hovering over it the agent says it is not logged in (Which is actually pretty accurate as we didn’t get the log in screen).

 

 

When right clicking the agent and choosing “Synchronize with Server” the agent seems to be functioning just fine. Even after a restart the login screen comes up as it should.

Unfortunately the problem came back after a couple of days so it seemed the “Synchronize with Server” was only a temporary fix. Systems would hang again at the blue blank screens.

Tricky part is that when troubleshooting this problem, you have to be careful to not trigger the UAC screen. Because one of the options in the UAC screen is to use the Imprivata Account, Imprivata automatically synchronizes with the appliance which triggers the temporary fix.

After doing a number of traces on some of the systems with the problems we noticed a strange error. The agent says “The system cannot find the file specified”.

But when going to that folder the files needed are found there as they should be.

This is strange as the agent keeps saying it can’t find those files.
It was no user rights issue as Imprivata was working before the issue occurred.
Before going any further we tried deleting the files on a normal functioning computer which resulted in the exact same problem. Which means the problem is related to these files.

As the operating system is Windows Embedded they are all using the build in Write Filter which should prevent any changes made to the system files.

According to the trace files made by Imprivata, one of the last actions before shutting down or rebooting the system is synchronizing with the appliance. This resulted in finding the cause of the problem.

Because Imprivata does a synchronize before shutting down it assumes that the agent is always in the lead of the shutdown or restart of the system. Although as mentioned earlier the systems are used in Computer on Wheels which uses battery packs enabling the user to walk around with the system. But when the battery starts running low there is a script that triggers a shutdown of the computer so that it shuts down cleanly instead of just pulling the power cord when the battery is empty.

However with this script that triggers the shutdown, Imprivata is not in the lead. This can result in a corruption of the .dat files in C:\ProgramData\SSOProvider\Offline\Global.
So next boot it will turn up without the Imprivata agent functioning.

At first we added a forced synchronize with the agent before the shutdown (C:\Program Files\Imprivata\Onesign Agent\ISXMenu.exe update). But as the Shutdown is forced the problem can still happen. So instead we copied the .dat files in the SSOProvider folder and put them in a safe place. After this we edited the shutdown script so it first copies the good files and overwrites the corrupted files in the SSOProvider folder before shutting down.

When this was added the Imprivata agent started functioning correctly again and the problem was solved.