PwnEscape Part 2: Escaping a Firecracker MicroVM by Hijacking a Live KVM vCPU Thread

Firecracker was designed to strip virtualization down to its bare essentials. Built by Amazon Web Services to back serverless platforms like AWS Lambda and AWS Fargate, the VMM discards legacy device emulation—no BIOS, no PCI bus enumeration, no sprawling QEMU device trees—and interfaces directly with Linux’s KVM through a compact Rust process. Less code in the monitor means a smaller attack surface for an untrusted guest to reach.

PwnEscape Part 2 targets a different layer entirely.

In the second installment of his PwnEscape research series, Paulos Yibelo of Pwn.ai takes a host physical memory read/write primitive established in Part 1 (CVE-2026-99999) and builds clean, reversible code execution inside the host Firecracker process on top of it. The technique avoids heap corruption and kernel structure spraying. Instead, it works backward through hardware memory management: locate the guest’s own Extended Page Table hierarchy in host RAM, recover the host Page Global Directory for the specific Firecracker process, defeat ASLR, find the saved pt_regs register frame of the active vCPU thread while it sits dormant inside KVM_RUN, and overwrite that saved state to redirect execution on the next VM exit. When the kernel returns from the ioctl, the host thread runs the injected code, the guest restores three modified stack words, and both the Firecracker monitor (PID 2936 in the recorded run) and the guest SSH session continue without interruption.

Why the KVM_RUN Return Path Is a Clean Target for Control-Flow Hijacking

Every Firecracker microVM corresponds to a single host process. Within that process, threads divide by role: an API server thread handles control-plane configuration over a UNIX domain socket, a VMM event-loop thread manages virtio network and block I/O, and one or more vCPU threads drive guest execution.

Each vCPU thread spends nearly its entire life inside a tight loop around one system call:

c

ioctl(vcpu_fd, KVM_RUN, 0);

When the vCPU thread calls this ioctl, execution crosses from host userspace (Ring 3) into the host kernel (Ring 0). At syscall entry, the kernel’s entry_SYSCALL_64 stub pushes the thread’s architectural state onto the top of its kernel stack as a struct pt_regs frame—21 quadwords, 168 bytes on x86-64. That frame records exactly what Firecracker expects to see when the ioctl eventually returns: the userspace instruction pointer pointing to the instruction right after the syscall in Firecracker’s ioctl wrapper, the userspace stack pointer, the system call number (RAX = 16 for SYS_ioctl), the pending return code slot (-ENOSYS, 0xffffffffffffffda), and the original arguments (RDI holding vcpu_fd, RSI holding 0xae80 for KVM_RUN, RDX set to 0).

With pt_regs safely on the kernel stack, KVM programs the VMCS and executes VMLAUNCH or VMRESUME, handing the physical CPU core to guest code. The host thread is effectively parked inside the kernel while the guest runs. It stays there until the guest does something requiring userspace intervention—writing a byte to the emulated UART port at 0x3f8, for example. That triggers a VM exit back to KVM, which sees that userspace device emulation is needed, unwinds the ioctl, pops the saved pt_regs off the kernel stack, and returns execution to host Ring 3.

The window the exploit uses is this: guest code runs on the physical CPU at the same moment the vCPU thread’s pt_regs sits dormant on the host kernel stack. A guest that can write to host physical memory can modify that saved register state before triggering an exit. When the exit fires, the kernel restores the frame and hands control of the host Firecracker thread to whatever address the guest placed in the saved RIP slot.

Navigating Host Physical Memory Without a Label in Sight

A host physical memory read/write primitive does not come with a map. Modern x86-64 hosts manage gigabytes of RAM where guest pages, host kernel stacks, and userspace process tables are scattered across arbitrary physical frames. Before touching a single byte, the exploit solves a four-stage address resolution problem through passive reads alone.

In the proof-of-concept environment—Firecracker v1.16.1 on a host running Linux 7.0.0-1011-aws with a guest running 6.18.44+—the Part 1 primitive exposes host physical addresses between 4 GiB and 192 GiB (FIRST_HPA_GIB = 4, LAST_HPA_GIB = 192). Inside the guest, that aperture is mapped into guest physical address space using 1 GiB huge-page aliases starting at L4_ALIAS_BASE (1ULL << 39, 0x8000000000, PML4 entry 1). Translating any HPA in that window to a Guest Physical Address mappable via /dev/mem is a bitmask operation, with one carve-out: HPA GiB 16 (MARKER_CONFLICT_HPA_GIB) conflicts with an existing range and is remapped to index 193 (MARKER_REMAP_ENTRY):

c

static uint64_t alias_entry_for_hpa_gib(uint64_t hpa_gib)
{
    return hpa_gib == MARKER_CONFLICT_HPA_GIB ?
           MARKER_REMAP_ENTRY : hpa_gib;
}
static uint64_t alias_gpa_for_hpa(uint64_t hpa)
{
    uint64_t hpa_gib = hpa >> 30;
    return L4_ALIAS_BASE |
           (alias_entry_for_hpa_gib(hpa_gib) << 30) |
           (hpa & (HUGE_1G_U64 - 1));
}

With the aperture open, the exploit runs four sequential resolution stages, passing validated physical and virtual pointers from one to the next.

Stage 1: Locating Guest Pages in Host RAM via Extended Page Table Traversal

The guest knows its own Guest Physical Addresses but has no information about where those pages sit in host physical memory. Under Intel VT-x, hardware translates GPAs to HPAs using Extended Page Tables—a four-level radix tree structurally identical to standard x86-64 page tables.

To anchor itself, the exploit allocates a page at a known GPA (0x08f03000 in the recorded run, tagged eventpath-v23-0907self02) and fills it with a unique byte pattern—a cryptographic nonce. The resolver then scans the exposed host physical aperture for candidate EPT root pages (EPTP PML4 tables). For each candidate root, it simulates a hardware EPT walk using the 9-bit table indices from bits 47:39, 38:30, 29:21, and 20:12 of GPA 0x08f03000. If all four levels resolve to a valid leaf, it reads the contents of that physical frame and compares them against the nonce:

python

for root in candidate_ept_roots:
    page_hpa = walk_ept(root, guest_gpa)
    if page_hpa is invalid:
        continue
    contents = read_host(page_hpa, length(nonce))
    if contents == nonce:
        candidates.append((root, page_hpa))

The initial scan flagged two candidate EPT hierarchies. Structural validation of the nested page table entries eliminated the false positive, leaving a single validated EPT root. Walking that root for GPA 0x08f03000 resolved to guest_page_hpa = 0x165610000. The exploit now has the exact host physical address of a page whose contents it controls.

Stage 2: Recovering the Firecracker Process’s Page Global Directory by Inverting Host Page Tables

Firecracker maps guest physical RAM directly into its own userspace virtual address space, typically via an mmap backing the guest memory region registered with KVM_SET_USER_MEMORY_REGION. That means the physical frame at 0x165610000 appears not only in KVM’s EPT tables, but also in a PTE inside the native x86-64 page tables of the Firecracker process itself.

Normal page table translation goes one way: given a process’s Page Global Directory (the physical address in CR3) and a 48-bit virtual address, you walk down four levels (PGD → PUD → PMD → PTE) to find the frame. The second resolver inverts this. It searches host physical memory for native page table structures with leaf entries pointing to guest_page_hpa (0x165610000), traces parent pointers upward through PMD and PUD to the top-level PGD root, and reconstructs the 48-bit virtual address from the 9-bit indices at each level.

To rule out unmapped remnants or kernel aliases, the resolver verifies each candidate bidirectionally:

python

for mapping in reverse_map_host_page_tables(guest_page_hpa):
    pgd = mapping.root
    guest_page_va = mapping.virtual_address
    resolved_hpa = walk_host_page_tables(pgd, guest_page_va)
    if resolved_hpa != guest_page_hpa:
        continue
    if read_host(resolved_hpa, length(nonce)) != nonce:
        continue
    process_candidates.append((pgd, guest_page_va))

This produced a single valid process address space rooted at host_pgd = 0x14afc2000. With host_pgd in hand, the exploit can translate any userspace virtual address inside the Firecracker process into a host physical address.

The C implementation of the native page table walker (walk_native) accounts for a few Linux kernel memory management behaviors that would otherwise produce false negatives: 1 GiB and 2 MiB huge pages (PTE_LARGE, bit 7), and NUMA balancing PROT_NONE leaf entries where the kernel temporarily clears the Present bit (0x1), sets bit 8 (0x100), and stores the bitwise-inverted physical frame number in the PTE ((~entry) & ADDRESS_MASK_4K).

Stage 3: Defeating ASLR and Validating ROP Gadgets Against Live Firecracker Memory

Knowing host_pgd means you can walk the virtual memory layout of the Firecracker process. Since Firecracker compiles as a Position-Independent Executable, host ASLR randomizes its base load address at every launch. The third resolver walks user-accessible virtual mappings inside host_pgd ((entry & (PTE_PRESENT | PTE_USER)) == (PTE_PRESENT | PTE_USER)), finds the mapped ELF header, and calculates the runtime load bias:

runtime_code_address = load_bias + code_address_in_elf

In the captured trace, the resolver recovered firecracker_base = 0x7298c2400000. Rather than trusting the ELF header, both the image resolver and the final C helper validate three specific instruction sequences in the .text segment (0x12000 to 0x276000) by reading their live bytes through host_pgd.

The write(2) wrapper (WRITE_VADDR = 0x27071e): At runtime virtual address 0x7298c267071e. The exploit checks a 15-byte prologue signature (48 83 ec 10 49 89 f2 48 89 d1 48 63 f7 6a 00):

asm

sub    rsp, 0x10
mov    r10, rsi
mov    rcx, rdx
movsxd rsi, edi
push   0x0

This internal wrapper takes standard System V AMD64 ABI arguments (RDI = fd, RSI = buf, RDX = count), shuffles them into the registers expected by the internal syscall dispatch, and calls write.

The KVM_RUN ioctl continuation (IOCTL_RETURN_VADDR = 0x26aad2): At runtime virtual address 0x7298c266aad2—the exact instruction where the ioctl(vcpu_fd, KVM_RUN, 0) wrapper resumes after the kernel returns to userspace. Its 13-byte signature (48 63 f8 e8 c3 e1 ff ff 48 83 c4 0x58 c3):

asm

movsxd rdi, eax          ; Sign-extend 32-bit ioctl return value into rdi
call   0x268c9a          ; Internal status/errno check helper
add    rsp, 0x58         ; Deallocate 88-byte stack frame
ret                      ; Return to Firecracker vCPU caller

The add rsp, 0x58 followed by ret matters for two reasons: it’s used to validate the userspace stack during frame discovery, and it determines where execution lands after the hijack completes.

The return-value cleanup gadget (ZERO_EAX_RET_VADDR = 0x1b274): At runtime virtual address 0x7298c241b274. A 3-byte sequence (31 c0 c3):

asm

xor    eax, eax
ret

Stage 4: Finding the Live pt_regs Frame on the Host Kernel Stack

With host_pgd (0x14afc2000) and firecracker_base (0x7298c2400000) confirmed, the fourth resolver scans host physical memory for the kernel stack holding the saved pt_regs of the active vCPU thread. Scanning gigabytes for a single register frame is noisy—dead kernel stacks, freed pages, and old context switches leave stale pt_regs structures scattered throughout RAM.

To ensure it only touches the live, in-flight vCPU thread, the frame finder enforces an eleven-field invariant across all 21 quadwords (PTREG_WORDS = 21) of the x86-64 struct pt_regs layout:

c

if (regs[10] != SYSCALL_AX_IN_PROGRESS || regs[12] != 0 ||
    regs[13] != KVM_RUN_REQUEST || regs[14] < 3 || regs[14] > 4096 ||
    regs[15] != SYS_IOCTL || regs[16] != ioctl_return_va ||
    regs[17] != USER_CS || !(regs[18] & UINT64_C(2)) ||
    regs[19] < UINT64_C(0x1000) || regs[19] >= (UINT64_C(1) << 47) ||
    regs[20] != USER_SS) {
    fprintf(stderr, "live KVM_RUN pt_regs validation failed\n");
    return 70;
}

In Linux’s x86-64 struct pt_regs, indices 0-14 hold general-purpose registers (r15, r14, r13, r12, rbp, rbx, r11, r10, r9, r8, rax, rcx, rdx, rsi, rdi), index 15 holds orig_rax, and indices 16-20 hold the hardware-pushed iret frame (rip, cs, eflags, rsp, ss). Each checked field maps to a concrete invariant of a live KVM_RUN call:

regs[10] (rax = 0xffffffffffffffda): When entry_SYSCALL_64 builds pt_regs, it initializes the saved rax slot to -ENOSYS (-38, 0xffffffffffffffda). This value stays in pt_regs->ax for the entire duration of an in-progress syscall and only gets overwritten with the actual return code right before the kernel exits to userspace. Matching SYSCALL_AX_IN_PROGRESS immediately filters out frames from completed calls.

regs[15] (orig_rax = 16): Must equal SYS_ioctl (16 on x86-64).

regs[14] (rdi = 3..4096): Holds vcpu_fd—a sensible non-stdio file descriptor integer.

regs[13] (rsi = 0xae80): Holds the ioctl request code. 0xae80 is _IO(KVMIO, 0x80), the exact constant for KVM_RUN.

regs[12] (rdx = 0): Third syscall argument, always 0 for KVM_RUN.

regs[16] (rip = load_bias + 0x26aad2): Must point to the exact byte after the syscall instruction inside Firecracker’s ioctl wrapper (0x7298c266aad2).

regs[17] (cs = 0x33) and regs[20] (ss = 0x2b): Standard 64-bit Linux userspace segment selectors, with the reserved architectural bit 1 set in regs[18] (eflags).

regs[19] (rsp): A canonical userspace pointer (0x1000 <= rsp < 1ULL << 47) that walks cleanly through host_pgd to a writable userspace stack page.

Stack frame caller check (rsp + 0x58): Because the ioctl continuation executes add rsp, 0x58; ret, the quadword at user_sp + 0x58 is the saved return address of the Firecracker function that called the ioctl wrapper. The resolver walks host_pgd to read that value and confirms it falls inside Firecracker’s .text segment (load_bias + 0x12000 to load_bias + 0x276000).

Applying these filters across host physical memory eliminated every stale frame and produced a single live structure at frame_hpa = 0x13a5bff48 (at offset 0xf48 within host physical page 0x13a5bf000).

Reversible Stack Geometry and the write(2) Payload

With host_pgd (0x14afc2000), firecracker_base (0x7298c2400000), and frame_hpa (0x13a5bff48) all confirmed, the exploit redirects the vCPU thread. The chosen payload is deliberately self-contained: force the Firecracker process to call its own write(2) wrapper to emit an 8-byte ASCII marker (“N8V23M2\n”) to stderr (fd = 2), pass back a 0 return value so the KVM_RUN loop sees nothing unusual, and return to the normal ioctl continuation path.

Calling write(2, “N8V23M2\n”, 8) requires a valid userspace pointer in RSI and a return address chain on Firecracker’s userspace stack. Both are staged on the vCPU thread’s private stack relative to the saved stack pointer user_sp (regs[19]):

marker_va = user_sp + 0x30 (MARKER_STACK_OFFSET): 48 bytes above the current user_sp, inside the active 88-byte stack frame of the ioctl wrapper. Temporarily overwritten with “N8V23M2\n”.

rop_stack_va = user_sp – 16 (ROP_STACK_OFFSET): 16 bytes below user_sp. Becomes the new stack top when write_va is entered. Overwritten with zero_eax_ret_va (load_bias + 0x1b274).

ioctl_return_slot_va = user_sp – 8: 8 bytes below user_sp, immediately above rop_stack_va. Overwritten with ioctl_return_va (load_bias + 0x26aad2).

Before writing a single byte, the PoC confirms that all three offsets sit on the same 4 KiB physical page (marker_hpa & ADDRESS_MASK_4K) and saves their original 64-bit values (original_marker, original_rop_first, original_rop_second). It then writes the marker string and two return addresses to the mapped stack page, issues a sequential-consistency memory barrier (__atomic_thread_fence(__ATOMIC_SEQ_CST)), and atomically rewrites five fields in the live pt_regs:

c

__atomic_store_n(&live_regs[14], UINT64_C(2), __ATOMIC_SEQ_CST);          // RDI = 2 (stderr)
__atomic_store_n(&live_regs[13], marker_va, __ATOMIC_SEQ_CST);            // RSI = &"N8V23M2\n"
__atomic_store_n(&live_regs[12], MARKER_LENGTH, __ATOMIC_SEQ_CST);        // RDX = 8
__atomic_store_n(&live_regs[19], rop_stack_va, __ATOMIC_SEQ_CST);         // RSP = user_sp - 16
__atomic_store_n(&live_regs[16], write_va, __ATOMIC_SEQ_CST);             // RIP = write wrapper

The xor eax, eax; ret gadget at user_sp – 16 addresses a specific problem. KVM_RUN normally returns 0 when it exits to userspace for device emulation. But the write wrapper returns 8 (the byte count) in RAX. If write returned directly to ioctl_return_va (movsxd rdi, eax; call 0x268c9a), Firecracker’s status checker would receive 8 instead of 0, interpret that as a fatal KVM_RUN loop violation, and abort the microVM. The two-instruction gadget zeros RAX and corrects stack pointer alignment before execution reaches the ioctl continuation.

Triggering the VM Exit, Executing the Payload, and Restoring State

Inside the guest, the PoC calls pwrite(port_fd, “!”, 1, 0x3f8), writing a single ‘!’ to the legacy COM1 I/O port (SERIAL_PORT = 0x3f8).

The I/O write causes an immediate VMEXIT (EXIT_REASON_IO_INSTRUCTION). KVM handles the exit, sees that port 0x3f8 belongs to Firecracker’s userspace device emulation, and returns 0 from kvm_arch_vcpu_ioctl_run. As the kernel exits SYS_ioctl back to userspace, it restores the modified pt_regs: RDI = 2, RSI = marker_va, RDX = 8, RSP = user_sp – 16, RIP = write_va.

Now running in host Ring 3 as the Firecracker vCPU thread, write_va calls write(2, “N8V23M2\n”, 8). The 8 bytes land on Firecracker’s stderr.

When write_va finishes, its final ret pops the top of the stack at user_sp – 16, jumping to zero_eax_ret_va and advancing RSP to user_sp – 8. The gadget runs xor eax, eax (RAX = 0) and ret.

That second ret pops ioctl_return_va from user_sp – 8 and advances RSP to its original value. Execution lands at 0x7298c266aad2 with RAX = 0 and RSP = user_sp—identical to a normal KVM_RUN return.

Firecracker processes the ‘!’ serial output, loops around, and calls ioctl(vcpu_fd, KVM_RUN, 0) again, resuming the guest right after the pwrite call. The moment pwrite returns inside the guest, the PoC copies original_marker, original_rop_first, and original_rop_second back into the three stack slots on stack_page and flushes the barrier.

A post-exploitation read-only validation pass (mode=”dry-run”) against frame_hpa = 0x13a5bff48 confirmed the result: exactly one 8-byte copy of N8V23M2\n on Firecracker’s stderr (SHA-256 5b8c0c0b5d5acb9ddf96891c79db8004832fd084e5e2d77484075995396fabbf), all three stack words at their original pre-exploit values, no .text or shared library mappings modified, zero host kernel errors, and the Firecracker process (PID 2936) still running the guest at 1615.72 seconds of uptime over an active SSH session.

What the Jailer Stops and What It Cannot Address

In the PwnEscape Part 2 test environment, Firecracker v1.16.1 ran directly as root via systemd-run, pinned to specific cores with taskset, without the production jailer. Under that configuration, hijacking the vCPU thread gives the guest immediate code execution as root in the host’s initial namespace.

Production multi-tenant deployments pair Firecracker with the jailer, which puts the process into an isolated chroot, enters restricted network and PID namespaces, drops privileges to an unprivileged UID/GID, and applies per-thread seccomp-bpf filters before KVM_RUN is ever called. Firecracker compiles distinct seccomp filters for its API, VMM, and vCPU threads, restricting the vCPU thread to a narrow syscall whitelist (ioctl, read, write, futex, sigaltstack, munmap, and a few others). Landing inside a jailed vCPU thread via pt_regs hijacking and calling execve(“/bin/sh”, …) would trip the seccomp filter and kill the process with SIGSYS.

But the jailer’s design doesn’t address two things the technique exposes.

Full process address space visibility. Even inside a seccomp-restricted thread, the attacker has host_pgd, which covers the entire Firecracker process address space—data structures belonging to the sibling VMM and API threads, all existing executable mappings, and anything else in the process. ROP chains can target any of it.

The host physical memory primitive itself. The vCPU hijack demonstrated here sits on top of a host physical memory read/write primitive. A guest that can walk EPT roots, traverse host PGDs, and locate kernel-space pt_regs frames at addresses like 0x13a5bff48 is already reading and writing host kernel memory from inside the guest. Replacing the pt_regs userspace redirect with a direct modification of the process’s kernel struct cred—identified through the same task/page-table resolution chain—or overwriting host kernel function pointers bypasses userspace seccomp and jailer namespaces entirely.

Why the Resolution Chain Is the Real Story

The write-to-stderr payload is a demonstration of control, not the substance of the research. The more interesting piece is the four-stage resolution chain that makes the hijack deterministic and reversible.

Hardware features like Extended Page Tables and randomized 64-bit address spaces are often treated as barriers that make physical memory corruption unreliable. The chain here works through them one step at a time. Using a known guest nonce to validate an EPT walk pins the exploit to the correct hierarchy out of everything in host RAM. Inverting the host page tables to recover the process CR3 identifies exactly whose address space you’re browsing. Calculating the PIE load bias from live memory removes ASLR as a variable. And filtering kernel stack frames against eleven simultaneous register and stack invariants guarantees you’re modifying the live frame, not a stale copy—with enough state information to restore the original values without leaving a trace.

Each stage is a deterministic read followed by a narrower search. By the time the exploit writes anything, it has already verified the target seven ways. The end result is a reversible function call from guest to host, and the process keeps running when it’s done.