KoroIRC Developer Blog

I hate skinned applications. Hate them, with a burning passion. The only thing I hate more are skinned applications that “try” to look native but spectacularly fail on anything else but the default theme.

Now, it ain’t so bad when applications restrict that to their client area. It is still useless and disrespectful of my settings, but at least it’s contained. But more and more apps take bolder steps at spitting in the face of my settings — hint: I set them that way for a reason, because I like it that way — and, sadly, Microsoft is no exception to this.

Windows Live Messenger is a big culprit in this case. Up until Wave 3, even though it defaulted to using custom window captions, a simple “Show the menu bar” would bring the regular ones back. But, as it seems to be the fashion these days, that possibility was removed. That, and they also now impose ClearType usage, even if you explicitly disabled it. Not that important now, since LCD screens are everywhere, but my old CRT monitor only broke 6 months ago.

But if you played a little with WLM9, you notice it reverts to regular window captions in certain cases, like if you maximize a window. A little poking around in WinDbg quickly shows that it’s just using window regions to do its bidding. That, along with a friend that showed me that under Aero Glass, it actually just lets the window manager do its job and uses regular captions, was more than enough to tip me off that something could be done very easily about this problem.

My first try was to try and block the SetWindowRgn call outright. I basically overwrite the code for that function in the process’ memory with the assembly equivalent of this:

int HookSetWindowRgn(HWND hWnd, HRGN hRgn, BOOL bRedraw)
{
    if (hRgn)
        DeleteObject(hRgn);
    return 1;
}

There are multiple problems with this approach. First, since WLM hides the whole caption and window borders, it has to handle WM_NCHITTEST in order to allow window resizing from within its client area. Since it also still thinks the caption and borders are hidden, it does not handle these cases, and as such, returns HTBOTTOM when the cursor is on the window caption. Also, the theming system in UxTheme.dll also calls this function — why it does not call NtUserSetWindowRgn directly is beyond me — in order to have those round window corners, and this “fix” disables that too.

My second and final solution came upon noticing that when WLM itself decides to show these captions, they work properly, and also the pseudo-caption inside the client area disapprears, too. If there was a way to trick WLM into thinking one of the “show the caption” conditions is met, it would by itself comply and show it without additional work. I first tried to hook the IsZoomed function directly to make it always return TRUE, but again it caused problems with UxTheme.dll.

I started by looking for calls to SetWindowRgn in all of WLM’s binaries. One of these files is UxCore.dll. Now, the fun part with exporting C++ classes or functions directly with __declspec(dllexport) is that all the decorated names are kept intact, which is very handy for reverse engineering like this since they contain full parameters and return type information. It did not take much time to find the UXFramelessManager class, which is responsible for hiding or showing the window caption, and handling everything that is associated with it. That class has a method called UpdateFrame, which does the work of determining if a window region needs to be set, and setting or clearing it as needed.

The UpdateFrame method checks that at least one of these three conditions is met:

• The window is maximized
• Aero Glass is enabled
• High Contrast Mode is enabled

If that is the case, it will set a NULL region for the window and not do any custom handling of the WM_NCHITTEST message. So, I figured, let’s make it think the window is always maximized! The code that does this check is:

7033e028  push  dword ptr [esi+8]
7033e02b  shr   eax,13h
7033e02e  and   al,1
7033e030  mov   byte ptr [ebp-1],al
7033e033  call  dword ptr [__imp__IsZoomed@4]

The instruction at 7033e028 pushes the HWND of the window on the stack. The next three instructions are interleaved code that does something else, probably due to agressive compiler optimization settings, then the call is done at 7033e033. So all we need to do is to change the code to just always assume IsZoomed returned TRUE, padding as appropriate with NOPs:

7033e028  nop
7033e029  nop
7033e02a  nop
7033e02b  shr   eax,13h
7033e02e  and   al,1
7033e030  mov   byte ptr [ebp-1],al
7033e033  mov   eax,1
7033e038  nop

All that was needed was to patch that new stream of instructions at the right file offset in UxCore.dll:

0003D428 -> 90 90 90 C1 E8 13 24 01 88 45 FF B8 01 00 00 00 90

Upon starting WLM, I had my window captions indeed:

(Yes, I went back to Windows Classic, despite my claims to the contrary.)

There is still a bug with this though. It it not apparent under Windows Classic, but UxTheme.dll causes problems again, albeit a minor one this time. Basically, UpdateFrame will still call SetWindowRgn on window creation with a NULL region, clearing the one set by UxTheme.dll. This is the code that does it:

7033e141  push   1
7033e143  push   ebx
7033e144  push   dword ptr [esi+8]
7033e147  call   dword ptr [__imp__SetWindowRgn@12]

Note that EBX is zeroed out at the beginning of the function, so this code is the equivalent of:

SetWindowRgn(m_hWnd,NULL,TRUE);

All we have to do is to replace that whole code sequence with NOPs again. I won’t present the code here, but directly the bytes to patch in the file:

0003D541 -> 90 90 90 90 90 90 90 90 90 90 90 90

Once this is patched, WLM will have regular window captions, with no glitch (that I found yet). It is sad though, that I had to resort to patching code for such a trivial matter, especially that older versions of WLM allowed this, so they actually had to remove code to make it not-work…

All code addresses and listings comes from UxCore.dll version 14.0.8050.1202 in the WLM installation directory. Make sure you have that exact same version before patching the file, of course.

Update: This also works with UxCore.dll version 14.0.8064.206, as none of the offsets have changed.

This entry was posted on Wednesday, January 14th, 2009 at 12:27 and is filed under Reverse Engineering. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.