For the full code see the git repo.

Introduction

This project aims to give a simple overview on how good various x64 hooking engines (on windows) are. I’ll try to write various functions, that are hard to patch and then see how each hooking engine does.

I’ll test:

• EasyHook
• PolyHook
• MinHook
• Mhook

(I’d like to test detours, but I’m not willing to pay for it. So that isn’t tested :( )

There are multiple things that make hooking difficult. Maybe you want to patch while the application is running – in that case you might get race conditions, as the application is executing your half finished hook. Maybe the software has some self protection features (or other software on the system provides that, e.g. Trustee Rapport)

Evaluating how the hooking engines stack up against that is not the goal here. Neither are non-functional criteria, like how fast it is or how much memory it needs for each hook. This is just about the challenges the function to be hooked itself poses.

Namely:

• Are jumps relocated?
• What about RIP adressing?
• If there’s a loop at the beginning / if it’s a tail recurisve function, does the hooking engine handle it?
• How good is the dissassembler, how many instructions does it know?
• Can it hook already hooked functions?

At first I will give a short walk through of the architecture, then quickly go over the test cases. After that come the results and an evaluation for each engine.

I think I found a flaw in all of them; I’ll publish a small POC which should at least detect the existence of problematic code.

A word of caution: my results are worse than expected, so do assume I have made a mistake in using the libraries. I went into this expecting that some engines at least would try to detect e.g. the loops back into the first few bytes. But none did? That’s gotta be wrong.

Another word of caution: parts of this are rushed and/or ugly. Please double check parts that seem suspicious. And I’d love to get patches, even for the most trivial things – spelling mistakes? Yes please.

Result