Rereading 0day Security (4), Early Ways Around DEP and ASLR

2624 words
13 minutes
Rereading 0day Security (4), Early Ways Around DEP and ASLR
This post is also available in 简体中文.

Part four of “Rereading 0day Security”, lab notes from 0day Security: Software Vulnerability Analysis Techniques, 2nd edition (by failwest and others), written in 2020 and tidied up in 2026. Every trick in the first three posts quietly assumed two things, that data pages can execute and that addresses are fixed. This post is about the two protections aimed at exactly those assumptions. DEP makes data pages non-executable, and ASLR stops addresses from being fixed. Then we look at how people got around them back then. The popup is still the shellcode from part one, and I won’t paste it all again. Every image these notes had back then is dead now, so words will have to do.

What DEP is#

The root of overflow attacks is that data and code aren’t kept apart, but redesigning computer architecture isn’t realistic, so all we can do is find ways to make it harder. DEP is one of them:

  • Memory pages holding data are marked non-executable, so when an overflow jumps into shellcode, the CPU raises an execution exception
  • There’s software DEP and hardware DEP. Software DEP is really just the SafeSEH machinery. Hardware DEP is called NX (No-Execute) on AMD and XD (Execute Disable) on Intel
  • The system sets an NX/XD bit for each page in the page tables, where 0 means executable and 1 means not

DEP has four modes:

  • OptIn: DEP only for system components and services, other programs aren’t protected unless the user marks them by hand. A program can turn this protection off at runtime. Mostly used on ordinary desktop systems
  • OptOut: DEP for everything except programs on an exclusion list. Mostly used on servers
  • AlwaysOn: DEP forced on for every program, can’t be turned off
  • AlwaysOff: rarely used

Fixing what I got wrong back then. The second mode is OptOut (I wrote “Output” at the time). Also, 64-bit processes always have DEP on and can’t turn it off, but that doesn’t mean “AlwaysOn is only for 64-bit”. AlwaysOn is a system-wide policy you can set on a 32-bit system, and it’s a different thing from “64-bit processes are forced to use DEP”.

On the program side, linking with /NXCOMPAT sets IMAGE_DLLCHARACTERISTICS_NX_COMPAT (0x100) in DllCharacteristics of the PE header’s IMAGE_OPTIONAL_HEADER, saying the program is DEP compatible.

The overall idea of beating DEP with ret2libc#

DEP stops you from “running your own code on a data page”. So don’t run your own code. Jump into system functions that already exist and are executable anyway. That’s ret2libc (Return-to-libc), where every step of the exploit is replaced by code that’s already sitting in a system lib. The hard part is that not every instruction address is free of 0x00, and chained jumps go wrong easily.

The book gives three workable routes:

  • Jump to ZwSetInformationProcess to turn DEP off for this process, then go run the shellcode
  • Jump to VirtualProtect to make the shellcode’s page executable, then run it
  • Jump to VirtualAlloc to get some executable memory, copy the shellcode there and run it

Turning DEP off with ZwSetInformationProcess#

A process’s DEP flags live in _KEXECUTE_OPTION inside KPROCESS:

Pos0 ExecuteDisable : 1
Pos1 ExecuteEnable : 1
Pos2 DisableThunkEmulation : 1
Pos3 Permanent : 1
Pos4 ExecuteDispatchEnable : 1
Pos5 ImageDispatchEnable : 1
Pos6 Spare : 2

With DEP on, ExecuteDisable=1. With it off, ExecuteEnable=1. Once Permanent=1, these flags are locked and can’t change. All we need is to set this value to 0x02 (ExecuteEnable=1) and DEP is off. The matching call is:

ULONG ExecuteFlags = MEM_EXECUTE_OPTION_ENABLE; // 0x2
ZwSetInformationProcess(
NtCurrentProcess(), // -1
ProcessExecuteFlags, // 0x22
&ExecuteFlags,
sizeof(ExecuteFlags) // 0x4
);
Note

Test setup: Windows XP SP3, DEP=OptOut, VC6, optimizations off, release.

Stitching this call together on the stack with ret2libc is the most twisty part of the whole post. The stack layout is roughly:

... overflow padding ...
52 E2 92 7C ; address of a "mov eax,1; ret" (it later checks whether al is 1)
96 73 1B 5D ; "push esp; pop ebp; retn N" to repair ebp
1E AD 17 5D ; raise esp by another 0x24 to make 0x30 of stack space
B4 C1 C5 7D ; jmp esp
90 90 90 90
24 CD 93 7C ; call ZwSetInformationProcess (turn DEP off)
90 90 E9 2D ; jmp to the real shellcode
FF FF FF 90

Here’s the idea. The first ret goes to mov eax,1. The call needs to read ebp, which the overflow already trashed, so push esp; pop ebp; retn N copies esp into ebp. But retn N adds N, leaving ebp smaller than esp, and calling a subroutine would then wreck the stack, so esp gets raised by 0x24 to make 0x30 of space. retn 0x24 returns to where ZwSetInformationProcess gets called. After DEP is off, leave; ret comes back to the jmp esp, which jumps to the junk data after it, and one last jump lands in the shellcode itself.

Changing permissions with VirtualProtect#

Instead of turning DEP off, change the shellcode’s page to executable. VirtualProtect(address, size, 0x40, some writable address), where the flNewProtect=0x40 argument is PAGE_EXECUTE_READWRITE. The arguments contain 0x00, so the trigger function switches from strcpy to memcpy. The stack layout starts with the ebp repair, then the VirtualProtect address, a return address, the four arguments, and finally the shellcode itself. Once the call finishes, the page is executable, and it returns to the start of the shellcode.

Getting new memory with VirtualAlloc#

LPVOID VirtualAlloc(LPVOID lpAddress, SIZE_T dwSize,
DWORD flAllocationType, DWORD flProtect);

First use VirtualAlloc to get some executable memory, then memcpy to copy the shellcode over, and when memcpy returns it goes straight to the start of the shellcode in the new memory. The stack layout is just VirtualAlloc’s four arguments followed by memcpy’s three, laid out in order.

Two routes I couldn’t finish#

  • Borrowing existing RWX memory. Sometimes a process has a region that’s readable, writable and executable, so you copy the shellcode in and hijack the flow. Sadly I couldn’t find such a region in the processes I tested back then, so this lab never got finished
  • Borrowing .NET sections. .NET files also have sections like .text that get mapped with execute permission, so you put the shellcode in the executable section of a .NET control and have the vulnerable ActiveX control jump there. The idea holds up, and the Java Applet route is similar, but both got stuck back then because I couldn’t find the right versions, so I never finished them

What ASLR is#

Every method so far needs a known jump address first, whether that’s jmp esp or the instructions ret2libc chains together. If you can’t beat the attack, dodge it. ASLR (Address Space Layout Randomization) stops using fixed base addresses at load time, which messes with shellcode trying to find things.

ASLR existed as an idea back in the XP days, but XP only did a simple randomization of the PEB/TEB and didn’t touch module base addresses. It only really started working in Vista. Like SafeSEH, it needs support from both the program and the system, though the program’s part isn’t strictly required. A program built with /dynamicbase (since VS2005 SP1) sets IMAGE_DLL_CHARACTERISTICS_DYNAMIC_BASE in its PE header, and that’s ASLR support.

  • Image randomization: when a PE is mapped into memory, its virtual load address is randomized. The address is picked when the system boots and only changes after a reboot. There’s a registry switch ...\Memory Management\MoveImages, where 0 disables it, -1 forces randomization (whether or not the image has the flag), and anything else is normal mode (only images with the flag get randomized)
  • Stack and heap randomization: the stack and heap bases are picked randomly when the program runs. Unlike images, this is decided when the program starts, so the same program gets different stack and heap bases on two runs. Back then I ran it twice each on XP and Vista to compare. On Vista the heap addresses were far apart, on XP they were identical
  • PEB/TEB randomization: worked badly in my tests. The PEB was randomized by only one byte, in a very regular pattern, and the TEB basically never changed

Gaps in ASLR#

Here’s the key thing. Image randomization only changes the high bits, and the low two bytes of every instruction address in a module stay the same. For example, if base 0x00401234 gets randomized to 0x67291234, the low 1234 is untouched. That leaves two openings.

Opening one, borrow a module without ASLR#

Lab setup: Windows Vista SP0, IE7, Adobe Flash Player 9.0.124 (the book wants 9.0.262; if you can’t find it, 9.0.124 works instead).

Flash9k.ocx doesn’t use ASLR, so its load base is fixed and you can find trampolines in it. The lab control has no GS and IE7 has DEP off, so all it takes is overwriting the return address and using jmp esp to get back to the shellcode on the stack. Finding a trampoline was a whole string of traps. Several jmp esp addresses, read as bytes, formed instructions that touched eax/ecx (like imul esp, [eax+0x1D], 0x30), and those registers didn’t point anywhere valid, with no good fix-up instruction to be found. The jmp esi candidates either contained 0x00, ended in retn, or threw an exception right away. What finally worked was the CALL ESP at 0x300942F2 (which also raises esp by 8 bytes, conveniently skipping the junk). The shellcode still ran after rebooting the system, so the ASLR bypass worked.

Opening two, partial overwrite#

Since the low two bytes aren’t randomized, overwrite only the low two bytes of the return address (off-by-word) and leave the high bits alone, and the trampoline always works. Attacking memcpy lets you land anywhere in 0xXXXX00000xXXXXFFFF, and attacking a str function in 0xXXXX01000xXXXX00FF. Find a jmp eax in the program, put the low word of its address in, get the buffer size right, and overflow.

Lab setup: Vista SP0, DEP=OptIn default, VS2008, optimizations off, GS off, /NXCOMPAT:NO, release. Build it, run it, and the popup appears, even after a reboot. This trick is still useful today. Modern ASLR has much more entropy, but the offset within a page (the low 12 bits) is never randomized, so partial overwrites are still a common move when you don’t have enough of an info leak.


Heap spraying, finding shellcode in a browser#

ASLR means you don’t know where the shellcode is. Heap spraying (Heap Spray) flips the problem around: fill memory up so that “pretty much any address” lands on shellcode.

Browser attacks often use the stack and the heap together. Triggering the bug gets you EIP, but browser memory is complicated and it’s hard to locate the complete shellcode directly. JavaScript on the page can allocate heap memory, though, so you use JS to lay the shellcode out on the heap. The move is to point EIP at 0x0C0C0C0C, then have JS allocate lots of heap memory and fill it with chunks of “a big run of 0x90 + shellcode”. JS usually allocates from low addresses up, so after more than 200MB, 0x0C0C0C0C is covered by one of the chunks. As long as it lands in that chunk’s NOP area, it slides down into the shellcode and runs.

var nop = unescape("%u9090%u9090");
while (nop.length <= 0x100000/2) { nop += nop; } // build a 1MB block of 0x90
// JS adds extra info to strings (32B heap header, 4B length, 2B terminator); subtract it so each chunk is exactly 1MB
nop = nop.substring(0, 0x100000/2 - 32/2 - 4/2 - 2/2 - shellcode.length);
var slide = new Array(); // back then I misspelled Array as Arrary here, fixed now
for (var i = 0; i < 200; i++) { slide[i] = nop + shellcode; }

Cover the heap with 200 of these 1MB chunks, and as soon as any chunk’s NOPs cover 0x0C0C0C0C, you’re in.

Heap spraying against ASLR#

Heap spraying and ASLR are a natural pair. ASLR randomizes module addresses, but the heap spray claims a fixed magic address, 0x0C0C0C0C, which module randomization doesn’t affect. On Vista + IE7, with the vulnerable ActiveX control from before, spray the heap until it fills everything around 0x0C0C0C0C, then overwrite the function’s return address with 0x0C0C0C0C. When it returns, it lands in the sprayed memory and slides over the NOPs into the shellcode. It still works after a reboot.


Vtable hijacking#

Last, one more place to strike that the earlier posts kept mentioning and kept putting off, C++ virtual functions. It gets around GS and the SEH protections alike, so it gets its own section.

  • In C++, member functions marked virtual are virtual functions, and their entry addresses all live in a virtual table (vtable)
  • When an object calls a virtual function, it first follows the object’s vtable pointer to the vtable, then takes the function address from the vtable and calls it
  • The vtable pointer is stored in the object’s memory, followed right after by its other member variables

So if a member variable inside the object (say an array) overflows, you get a chance to change the vtable pointer, or a function pointer in the vtable, and the next virtual call jumps into the shellcode:

class Failwest {
public:
char buf[200];
virtual void test(void) { cout << "Vtable::test()" << endl; }
};
Failwest overflow, *p;
void main(void) {
char * p_vtable = overflow.buf - 4; // the vtable pointer comes before the member variables
// point the vtable pointer at a spot inside buf that we control
p_vtable[0] = 0x5C; p_vtable[1] = 0xBB;
p_vtable[2] = 0x40; p_vtable[3] = 0x00;
strcpy(overflow.buf, shellcode);
p = &overflow;
p->test(); // jumps into the shellcode
}

The object’s memory is on the heap, but this is a continuous, linear overwrite, so strictly speaking it’s neither a stack overflow nor a heap overflow. “Array overflow” or “contiguous overwrite” is a more accurate name. Using a DWORD SHOOT to change the vtable is often simpler. Vtable hijacking dodges GS (it fires before the function returns) and SafeSEH/SEHOP (neither one cares about vtables), so once GS was everywhere, it became a very popular route. That’s also exactly why it’s the main target of the modern protections below.


Looking at it again today#

Both protections in this post are still here, and they’re where modern exploitation really begins. Attackers and defenders have just both moved a long way forward.

On the DEP side, ret2libc grew straight into ROP (Return-Oriented Programming). The old “jump into one or two system functions” stitching became “chain hundreds or thousands of little snippets ending in ret (gadgets)”, which is enough to do any logic you want without running a single byte of new code, like calling VirtualProtect to make the real payload executable. The turn-off-DEP and make-it-executable tricks above are the most common way a ROP chain ends. Microsoft’s answer is CFG/XFG (restricting where indirect calls can go, aimed squarely at vtable hijacking like the one above), ACG (once a code page is executable it can’t be changed again, which blocks “change the permissions yourself, then write code”) and CIG (only images signed by Microsoft may load).

On the ASLR side, the biggest change is that an info leak became the first step of an exploit. Modern Windows forces ASLR by default, and 64-bit has high-entropy ASLR on top, so guessing addresses basically doesn’t work. Almost every full exploit chain today first needs a memory leak bug to get the real address of some module or heap, and only then works out where the gadgets and shellcode are. And both of the gaps in this post are still alive. The “offset within a page isn’t randomized” that partial overwrites rely on is still true today. The idea of spraying the heap to claim a fixed magic address isn’t dead either, though browsers added isolated heaps and put randomization and permission limits on JIT memory, so the old-style spray with a plain NOP slide doesn’t work as well as it used to. These days it’s more often paired with bugs like type confusion and UAF for precise heap layout.

None of the tricks in these four posts can be copied as-is today. But understanding them properly is exactly the groundwork for reading modern ROP, CFG and “leak first, then exploit”.

Share Article

If this article helped you, please share it with others!

Rereading 0day Security (4), Early Ways Around DEP and ASLR
https://blog.lumysia.net/posts/reread-0day-4-dep-and-aslr/
Author
Livia
Published at
2020-11-19
Related PostsSmart
1
Rereading 0day Security (1), Stack Overflows and Shellcode Basics
My first round of labs from the book 0day Security, covering PE address conversion, overwriting a return address with a stack overflow, the jmp esp trampoline, finding kernel32 through the PEB, encoding shellcode, plus a little format string bug at the end.
2
Rereading 0day Security (2), the NT5 Heap, DWORD SHOOT and the First Heap Protections
Debugging the free lists and lookaside lists on Windows 2000, using unlink for a DWORD SHOOT that overwrites a function pointer in the PEB, then a look at safe unlink, heap cookies and metadata encoding from XP SP2 on.
3
Rereading 0day Security (3), SEH Hijacking, Plus GS, SafeSEH and SEHOP
From overwriting SEH to hijack exception handling, to the GS cookie, SafeSEH's RtlIsValidHandler decision tree and SEHOP's chain integrity check, and how each one gets bypassed.
4
A Short History of Windows Exploit Mitigations (2000–2026)
Taking that old XP→Win7 table of GS, DEP, ASLR, SafeSEH and SEHOP all the way to Win11, adding CFG, XFG, the CET shadow stack, ACG, CIG, VBS/HVCI and KDP. Every generation of defense answers the attacks of the one before.
5
Taking Apart a Cracked iMazing Package That Defender Flagged, SFX, Batch Files and AutoIt Obfuscation
Peeling the TNT cracked iMazing package layer by layer, a WinRAR self-extractor, encrypted batch files, an AutoIt runner and a decompiled obfuscated script, until the two obfuscated strings finally decode and show what it's really doing.
Random PostsRandom

Comments

Table of Contents