Anti-reverse-engineering software: why static blacklists aren't enough

Most anti-reverse-engineering software starts with a list: if a well-known debugger is running, close the application. Against casual curiosity that works, and it costs almost nothing. Against anyone willing to rename a file or use a tool that isn't on the list, it does very little.

This guide explains what process and window blacklists are good for, why they can only ever catch known tools, and what behavioural checks and AI-assisted detection add on top. It finishes with a response policy that protects your product without punishing the customers who paid for it.

What static blacklists do well

A static blacklist compares what is running on the machine with a list of known analysis tools. It can match on several things: process names, window titles and window class names. When there is a match, the application reacts.

• –They are cheap. A list lookup costs almost nothing at runtime.
• –They are predictable. The same input always gives the same result, which makes them easy to reason about.
• –They catch the obvious cases. Plenty of attempts use public tools under their default names.
• –They can be updated remotely. In qPapel Auth, Remote Settings let you block window titles, class names, active processes and debugging tools without compiling a new build, and target rules by app version, country or user group.

Those strengths are why a blacklist belongs in every protection setup. The mistake is treating it as the whole setup.

Why process and window blacklists only catch known tools

A blacklist knows names, and only names. That gives it three blind spots that no amount of list maintenance can close.

There is also a failure in the other direction. A list that tries to cover more ground by matching broadly starts to catch legitimate software. Close the application on a paying customer who happens to have a developer tool open, and you've created a support ticket and possibly a refund, while a determined attacker simply works around the list.

What anti-reverse-engineering software has to cover

Analysing a desktop product usually involves a few broad activities: looking at the files, studying the code, watching the program while it runs, observing what it sends over the network, and finally changing it. Each activity calls for a different defence, and a blacklist only touches one of them.

The table also shows why the defences reinforce each other. Detection at runtime is far more effective when the binary contains nothing valuable, and server-side data is far safer when the session that receives it is being watched.

Two rows deserve a closer look because they are where most of the value lies: keeping the prize off the client, and noticing analysis however it is done. The first is covered in depth in client-side protection vs server-side secrets. The rest of this article is about the second.

What behavioural detection adds

Behavioural checks don't ask what a tool is called. They ask what is happening to your process and the machine it runs on. A renamed tool still has to do its job, and doing that job leaves signs.

• –Debugger presence. qPapel Auth can detect an attached debugger and ban the offending license. qPAnalyzer adds checks such as hardware breakpoints, process debug flags and kernel debugger presence.
• –Virtual machines. Anti-VM detection notices the virtual environments commonly used to analyse software.
• –Modified code. auth.CheckMemoryIntegrity() checks whether the program's memory has been modified at runtime, for example by instruction patching. auth.CheckTextIntegrity() adds CRC, session-based and patch checks.
• –Session behaviour. Session Token Enforcement lets you set maximum active session durations and lock sessions to IP addresses from Remote Settings.

// Illustrative pseudocode
void OnPremiumFeatureOpened() {
    auth.CheckMemoryIntegrity();   // has running code been modified?
    auth.CheckTextIntegrity();     // CRC, session and patch checks
    LoadFeatureFromServer();       // server data only after checks pass
}

Placement matters too. A check that runs once at start-up only says the program was clean at start-up. Running integrity checks again at the moments that matter, such as when a premium feature opens or before server data is requested, gives a much more useful picture of the whole session.

The important design choice is what happens after a check. A failed check that only shows a message invites someone to remove the message. A failed check that means the server stops releasing data, and the key is flagged, has consequences that can't be removed from the client.

What AI-assisted detection adds

Behavioural checks catch what analysis does to your process. They don't tell you which tools are open on the machine if those tools are careful. That is the gap AI-assisted detection fills.

qPAnalyzer inspects UI layout patterns, window handles and memory signatures, so it can recognise a tool regardless of cosmetic changes such as a new name. When a window looks suspicious, it captures a screenshot and the analysis engine reviews the visual interface, then assigns a calibrated threat score from 0 to 100%. Analysis tools look like analysis tools, whatever the executable is called.

Scoring matters as much as recognition. A yes-or-no verdict forces you to choose between missing real threats and acting on weak evidence. A score from 0 to 100% lets you treat a faint resemblance differently from a clear match, and lets you tighten or relax your response without touching the detection itself.

Just as important is what qPAnalyzer does to avoid false positives. Each product has its own whitelist with custom instructions for qPAnalyzer, findings are filtered per product in the Threat Vault, and a one-click "Not a threat" action removes anything that turns out to be harmless. There is a detailed walkthrough in how qPAnalyzer detects renamed and custom tools.

A proportionate response policy

Detection without a clear policy produces either noise or angry customers. The aim is to respond firmly to clear signals, carefully to uncertain ones, and to keep evidence for both.

1. 01

   Known hostile tool

   A blacklist match on a well-known analysis tool can act automatically: close the session and log it. Repeat matches on the same key justify a ban.

2. 02

   Unfamiliar tool, high score

   Let qPAnalyzer's score drive the action. PapelShip's risk engine combines signals into one score with warn, restrict, close and ban thresholds you define.

3. 03

   Unfamiliar tool, low score

   Log and review. If it is legitimate software your audience uses, add it to the product whitelist or mark it "Not a threat".

4. 04

   Use delays

   With delayed action mode, penalties trigger after a randomised delay, so the response doesn't point straight at the check that caused it.

5. 05

   Act centrally

   Ban the key and, where appropriate, the device. Because the decision is on the server, it applies at the next check without a new release.

Think about the honest customer's side of a closure as well. In the EasyAuth library, the application shows a "Security Policy Closing" message when it closes because of a policy violation, and includes extra information to help identify the cause. When a genuine customer contacts support, that information lets you tell a false positive from a real problem quickly, whitelist what needs whitelisting and get them running again.

Keeping detection current without shipping builds

The other weakness of a hard-coded blacklist is that it ages. A list compiled into your binary is out of date the day after release. Keeping the rules on the server avoids that.

• –Remote Settings change blacklists, anti-VM, integrity checks, debugger bans, the session system and HWID lock from the dashboard, with no client update.
• –Auto Rules let you automate actions with your own rules. Set them once and they run on their own.
• –qPAnalyzer updates improve detection on PapelShip's side. Recent releases expanded coverage and reduced false positives.

Server-side rules also make it safer to experiment. Because Remote Settings can target rules by app version, country or user group, you can try a new blacklist entry or a stricter threshold on a small group first, watch the results on the Computers page, and widen it only once you're confident it isn't catching honest customers.

If you want to go further, updating protection remotely without rebuilding covers the workflow in detail, and the qPAnalyzer product page shows the dashboard views.

Build protection that a new name can't get past

A static blacklist is a sensible first filter and a poor last line of defence. Behavioural checks notice what analysis does to your process. AI-assisted detection notices tools by what they are. Server-side decisions make sure that whatever is detected has consequences everywhere at once.

Keep the list, add the layers behind it, and set a response policy that is firm with clear signals and fair with uncertain ones. To see how these layers sit next to licensing and HWID binding, read a software protection stack for C++ and C#.