I was troubleshooting an unattended installation of a particular application. The install seemed to hang right away so I figured it was presenting some kind of message (error?).
Using a Window Spy tool I made the setup process visible and saw the following message:
The unattended install was supplied by the vendor and apparently they use InstallAware.
The setup tries to create a Tray Icon, probably a setup progress indicator, but this fails because there is no shell running (the installation is pushed from a deployment server).
The setup.exe extracts a bunch of files, including the actual installer executable and places this in a temp folder. Using Process Explorer I tracked down the path:
I opened the setup executable in Ida Pro and searched for the string "Cannot create shell notification icon" on the Strings window:
Then I checked where in the code this string is referenced (doubleclick on the string and press Ctrl-X):
From the Disassembly we can see that sub_4C3F0C is called and if this returns a value > 0 (Boolean TRUE) we jump to loc_4C39F6. if the return value is 0 the error message is displayed:
sub_4C4F0C makes a call into the Shell_NotifyIcon API:
I decided to patch the code by replacing the call to Shell_NotifyIcon with "return TRUE". I have 6 bytes to do this:
To return TRUE we need to set the EAX CPU register (which holds the return value) to 1. If I would use mov eax, 0 this would take up 5 bytes. To get the same result in less bytes we can xor eax with itself (value becomes 0) and the increment it with 1.
Finally we return with retn 8 (8 because the function takes two arguments which are both 4 bytes in a 32 bit application):
Now the installation continues without errors (screen belows shows the non silent installation):
Was once an enthusiastic PepperByte employee but is now working elsewhere. His blogs are still valuable to us and we hope to you too.