When you want to run a program, for example calc.exe, and the Antivirus (AV) doesn’t recognise it (for example with something you have just downloaded), it will be scanned. This is called static analysis because the program is ‘static’ – it isn’t executing (doing anything) yet. Bypassing detection from this is usually trivial, assuming your AV doesn’t use an emulator, because, as it turns out, malware often looks like normal software. Take, as an analogy, a .zip file; unless it has a particularly revealing name, you have no idea what is inside of it unless you open it. If your AV does use an emulator (a ‘fake computer’ sandbox in which it runs the software first) then static analysis can be more effective, but that’s a story for another time. Quick bit of terminology: when a program runs, it is called a process.
The other type of scanning is called dynamic analysis, which is where most malware is found and terminated. In the case where your AV can’t detect anything evil using static analysis, your AV will scan the process in question while it is running. There’s an obvious downside to this approach: malware might be able to do some damage before the AV catches it, which is why AVs will implement static analysis first. A common analysis method, used both in static and dynamic analysis, is to check whether the program / process has similar features to known malware, for example you could make a keylogger that had similar code to an already existing one.
An Antivirus can’t scan all parts of all processes all the time; that would have an unbearable performance impact. Instead of periodically scanning memory, Antiviruses tend to guess what the most likely places to contain malicious code are and scan them. A good place to look for malicious code is in RX / RWX regions in processes (particularly if they aren’t backed by an image).
What does this mean? Memory is allocated in chunks (regions) which have permissions assigned to them. For example, if you wanted to execute code in a memory region you allocate, you would assign it the X (execute) permission. If you needed to read data from it, you use R (read) and to write / change data, W (write). Code regions must at least have RX to be able to execute code (R because you need to read the instructions before executing them).
AVs typically will only scan a memory region if some code in it performs a suspicious Windows API call.
Typically, a process, in this case evil.exe, will perform a blacklisted API call, which will cause the AV to perform a scan of its executable regions. A potential, highly likely blacklisted function would be the undocumented NtCreateRemoteThread, which is used to inject code into another process, in this case the legitimate notepad.exe. The process evil.exe calls this function, which is intercepted by the AV driver and sent to the AV, which then inspects the parameters and decides whether to scan the process. If it does scan the process, and if it finds any malicious code, it will terminate evil.exe, otherwise it will let the API call continue. The monitoring itself cannot be easily subverted.
This security model is subject to a vital flaw: if a process can modify the contents of its executable regions to look legitimate, then (almost) whenever the AV scans the process, it will only ever find harmless looking memory and our malware will lie undetected. The AV may, very occasionally, randomly scan the process, but there’s a solution for that too: minimise the chance it will scan when malicious code is exposed by implementing the same modification technique on sleep functions, which will be called after the malware has performed an action, for example the malware could sleep for 10 seconds, check for an update from the malware author’s server, and repeat.
An implementation of this with one API call might look like the diagram to the right. The API call in question is hooked by the process itself, and when it calls it, it redirects execution to change the memory permission to RW.
An implementation I made follows. Ignore the ntdll references; I injected this into a system module for testing.
Functions are referenced often using indirect addressing, so to hook them, you can just change the direct (intermediate) reference.
For each blacklisted API, your program would create an entry in this table, which can be easily dynamically grown.
<Explanation + diagram of stack in x64 calling convention here>
<also: above purple scribbles will be in nice boxes similar to other diagrams>
A final, high-level implementation could look like this:
The MEM_FREEZE and MEM_UNFREEZE macros are a simplification of how the RW / R(W)X memory technique works; you can assume code inside is automatically isolated.
The function unpacked_main would be called after the process has unpacked itself to avoid static detection. Malicious code, that would be detected by an AV, is present and detectable in the process, but only very briefly in the program loop. Any time it calls a function that is blacklisted, or it sleeps, it won’t be detected. This could be made even better by implementing MEM_FREEZE and MEM_UNFREEZE around the update request, which could take a second, as opposed to the other logic taking <1ms, which would leave only a minuscule detection window open.
Or, you know, you could just ask whoever you’re hacking to turn off their Antivirus.