Question

Is it possible to define a C++ function where one parameter is passed via the EAX register?

An old game doesn't work on my PC and I want to fix it. A function of the game might cause the problem. So I decided to hook this function with Detours. The following code is generated by IDA. mbstowcs is the function I want to hook.

.text:0043A9D4 ; int __fastcall mbstowcs(LPWSTR lpWideCharStr, LPCSTR lpMultiByteStr)
.text:0043A9D4 mbstowcs        proc near               ; CODE XREF: sub_408D58+39↑p
.text:0043A9D4                                         ; sub_408D58+65↑p ...
.text:0043A9D4
.text:0043A9D4 cchWideChar     = dword ptr -14h
.text:0043A9D4
.text:0043A9D4                 push    ebx
.text:0043A9D5                 push    esi
.text:0043A9D6                 push    edi
.text:0043A9D7                 push    ebp
.text:0043A9D8                 push    ecx
.text:0043A9D9                 mov     [esp+14h+cchWideChar], ecx
.text:0043A9DC                 mov     esi, edx
.text:0043A9DE                 mov     edi, eax
.text:0043A9E0                 test    edi, edi
.text:0043A9E2                 jz      short loc_43A9F1
.text:0043A9E4                 cmp     [esp+14h+cchWideChar], 0
.text:0043A9E8                 jnz     short loc_43A9F1
.text:0043A9EA                 xor     eax, eax
.text:0043A9EC                 jmp     loc_43AA77
.text:0043A9F1 ; ---------------------------------------------------------------------------
.text:0043A9F1
.text:0043A9F1 loc_43A9F1:                             ; CODE XREF: mbstowcs+E↑j
.text:0043A9F1                                         ; mbstowcs+14↑j
.text:0043A9F1                 test    edi, edi
.text:0043A9F3                 jz      short loc_43AA5D
.text:0043A9F5                 mov     ebx, [esp+14h+cchWideChar]
.text:0043A9F8                 push    ebx             ; cchWideChar
.text:0043A9F9                 push    edi             ; lpWideCharStr
.text:0043A9FA                 push    0FFFFFFFFh      ; cbMultiByte
.text:0043A9FC                 push    esi             ; lpMultiByteStr
.text:0043A9FD                 push    9               ; dwFlags
.text:0043A9FF                 push    0               ; CodePage
.text:0043AA01                 call    MultiByteToWideChar
.text:0043AA06                 test    eax, eax
.text:0043AA08                 jz      short loc_43AA0D
.text:0043AA0A                 dec     eax
.text:0043AA0B                 jmp     short loc_43AA77

At line .text:0043A9DE, a parameter seems to be passed via the EAX register. As far as I know, the target function and detour function must have same calling convention. The problem is, I don't know what calling convention requires that parameter be passed via EAX.

I'm using MSVC 19.40.

I tried to use the default and fastcall calling conventions, but no parameters are passed via EAX.

How to hook this function? How to define a function whose parameter is passed through EAX?

 2  112  2
1 Jan 1970

Solution

 2

It works. I defined a function like this:

size_t __attribute__((regparm(3))) better_mbstowcs(wchar_t* pwcs, const char* s, size_t n)

and compiled it with clang-cl. IDA output:

.text:10001060                 public better_mbstowcs
.text:10001060 better_mbstowcs proc near               ; DATA XREF: DllMain(x,x,x)+3C↓o
.text:10001060                                         ; DllMain(x,x,x)+69↓o ...
.text:10001060                 push    esi
.text:10001061                 test    eax, eax
.text:10001063                 jz      short loc_1000106B
.text:10001065                 xor     esi, esi
.text:10001067                 test    ecx, ecx
.text:10001069                 jz      short loc_10001088
.text:1000106B
.text:1000106B loc_1000106B:                           ; CODE XREF: better_mbstowcs+3↑j
.text:1000106B                 test    eax, eax
.text:1000106D                 cmovz   ecx, eax
.text:10001070                 push    ecx             ; cchWideChar
.text:10001071                 push    eax             ; lpWideCharStr
.text:10001072                 push    0FFFFFFFFh      ; cbMultiByte
.text:10001074                 push    edx             ; lpMultiByteStr
.text:10001075                 push    0               ; dwFlags
.text:10001077                 push    0               ; CodePage
.text:10001079                 call    ds:MultiByteToWideChar

It looks good.

2024-07-22
panel