The fundamental problem with usermode-only anti-cheat is the trust model. A usermode process runs at ring 3, subject to the full authority of the kernel. Any protection implemented entirely in usermode can be bypassed by anything running at a higher privilege level, and in Windows that means ring 0 (kernel drivers) or below (hypervisors, firmware).
The kernel was, for a long time, the exclusive domain of cheats. Kernel-mode cheats could directly manipulate game memory without going through any API that a usermode anti-cheat could intercept. They could hide their presence from usermode enumeration APIs trivially. They could intercept and forge the results of any check a usermode anti-cheat might perform.
The escalation has been relentless. Usermode cheats gave way to kernel cheats. Kernel anti-cheats appeared in response. Cheat developers began exploiting legitimate, signed drivers with vulnerabilities to achieve kernel execution without loading an unsigned driver (the BYOVD attack). Anti-cheats responded with blocklists and stricter driver enumeration. Cheat developers moved to hypervisors, running below the kernel and virtualizing the entire OS. Anti-cheats added hypervisor detection. Cheat developers began using PCIe DMA devices to read game memory directly through hardware without ever touching the OS at all. The response to that is still being developed.
effective kernel anti-cheat requires the same OS primitives that malicious kernel software uses, because those primitives are what provide the visibility needed to detect cheats. Any sufficiently capable kernel anti-cheat will look like a rootkit under static behavioral analysis, because capability and intent are orthogonal at the kernel API level. This is a constraint imposed by Windows architecture, not a design choice unique to any particular anti-cheat vendor.
Modern kernel anti-cheats universally follow a three-layer architecture: [kernel driver, usermode service, and game-injected DLL.]
The separation of concerns here is both architectural and security-motivated. The kernel driver can do things no usermode component can, but it cannot easily make network connections or implement complex application logic. The service can do those things but cannot directly intercept system calls. The in-game DLL has direct access to game state but runs in an untrustworthy ring-3 context.
IOCTLs (I/O Control Codes) are the primary communication mechanism between usermode and a kernel driver.
Named pipes are used for IPC between the service and the game-injected DLL. A named pipe is faster and simpler than routing everything through the kernel, and it allows the service to push notifications to the game component without polling.
Shared memory sections created with NtCreateSection and mapped into both the service process and the game process via NtMapViewOfSection allow high-bandwidth, low-latency data sharing.
Vanguard loads vgk.sys at system boot. The driver is configured as a boot-start driver (SERVICE_BOOT_START in the registry), meaning the Windows kernel loads it before most of the system has initialized. This gives Vanguard a critical advantage: it can observe every driver that loads after it. Any driver that loads after vgk.sys can be inspected before its code runs in a meaningful way. A cheat driver that loads at the normal driver initialization phase is loading into a system that Vanguard already has eyes on.
Vanguard maintains an internal allowlist of drivers that are permitted to co-exist with a protected game. Any driver not on this list, or any driver that fails integrity checks, can result in Vanguard refusing to allow the game to launch. This is an allowlist model rather than a blocklist model, which is architecturally much stronger.
ObRegisterCallbacks is perhaps the single most important API for process protection. It allows a driver to register a callback that is invoked whenever a handle to a specified object type is opened or duplicated. For anti-cheat purposes, the object types of interest are PsProcessType and PsThreadType.
When a cheat calls OpenProcess(PROCESS_VM_READ | PROCESS_VM_WRITE, FALSE, gameProcessId), the anti-cheat’s ObRegisterCallbacks pre-operation callback fires. The callback checks whether the target process is the protected game process. If it is, it strips PROCESS_VM_READ, PROCESS_VM_WRITE, PROCESS_VM_OPERATION, and PROCESS_DUP_HANDLE from the desired access. The cheat receives a handle, but the handle is useless for reading or writing game memory. The cheat’s ReadProcessMemory call will fail with ERROR_ACCESS_DENIED.
PsSetCreateProcessNotifyRoutineEx allows a driver to register a callback that fires on every process creation and termination event system-wide.
Anti-cheats use this callback to detect cheat tool processes spawning on the system. If a known cheat launcher or injector process is created while the game is running, the anti-cheat can immediately flag this. Some implementations also set CreateInfo->CreationStatus to a failure code to outright prevent the process from launching.
PsSetCreateThreadNotifyRoutine fires on every thread creation and termination system-wide. Anti-cheats use it specifically to detect thread creation in the protected game process. When a new thread is created in the game process, the callback fires and the anti-cheat can inspect the thread’s start address.
A thread created in the game process whose start address does not fall within any loaded module’s address range is a strong indicator of injected code. Legitimate threads start inside module code. An injected thread typically starts in shellcode or manually mapped PE code that has no module backing.
Anti-cheats periodically hash the code sections (.text sections) of the game executable and its core DLLs. A baseline hash is computed at game start, and periodic re-hashes are compared against the baseline. If the hash changes, someone has written to game code, which is a strong indicator of code patching (commonly used to enable no-recoil, speed, or aimbot functionality by patching game logic).
When a legitimate DLL loads, it appears in the process’s PEB module list, in the InLoadOrderModuleList, and has a corresponding VAD_NODE entry with a MemoryAreaType that indicates the mapping came from a file. Manual mapping bypasses the normal loader, so the mapped code appears in memory as an anonymous private mapping or as a file-backed mapping with suspicious characteristics.
The primary defense against BYOVD is a blocklist of known-vulnerable drivers. The Microsoft Vulnerable Driver Blocklist (maintained in DriverSiPolicy.p7b) is built into Windows and distributed via Windows Update. Anti-cheats maintain their own, more aggressive blocklists.
A PCIe DMA (Direct Memory Access) cheat uses a PCIe-connected device - typically a development FPGA board - that can directly read the host system’s physical memory via the PCIe bus without the CPU being involved. The pcileech framework and its LeechCore library provide the software stack for these devices. The device physically appears on the PCIe bus, acquires access to the host’s physical memory via the PCIe TLP (Transaction Layer Packet) protocol, and reads game process memory by translating virtual addresses to physical addresses (using the page tables, which are also in physical memory and can be read by the device).
The attacking machine (running the cheat software) is physically separate from the victim machine (running the game). All cheat logic runs on the attacker’s machine. The game machine has no processes, no drivers, no memory allocations from the cheat. From a pure software perspective, the game machine is completely clean.
With IOMMU enabled, a DMA device that is not associated with any OS driver has no IOMMU mappings and cannot access arbitrary physical memory. This is theoretically a hardware-level defense against DMA attacks.
In practice, the IOMMU defense has significant gaps. Many gaming motherboards ship with IOMMU disabled by default. Even when enabled, the IOMMU configuration is complex and many systems have misconfigured IOMMU policies that leave large physical memory ranges accessible. And critically, DMA firmware that successfully impersonates a legitimate PCIe device (e.g., a USB controller or a network card that the OS has granted IOMMU access to) can potentially access memory through the IOMMU using the legitimate device’s granted permissions.
Aimbot detection targets the statistical properties of mouse movement. Human aiming exhibits specific properties: Fitts’ Law governs approach trajectories, there is characteristic deceleration as the cursor approaches a target, velocity profiles have specific acceleration and deceleration curves, and there is measurement noise. An aimbot performing perfect linear interpolation to a target produces movement that violates these properties. A triggerbot (which fires automatically when the crosshair is on target but does not manipulate mouse movement) is detected via reaction time analysis: human reaction times to a target crossing the crosshair have a minimum physiological floor (approximately 150-200ms) with a characteristic distribution. Reaction times below this floor with high consistency indicate automation.
Cheat developers sometimes use nested hypervisors to create a transparent analysis environment: they run the game in a VM, with the cheat running in the VM’s host. Detection of nested hypervisors relies on timing anomalies: CPUID executed inside a nested VM is handled by two hypervisors in sequence, introducing double the overhead. RDMSR and WRMSR instructions similarly have amplified latency. Statistical analysis of hundreds of timing measurements can reliably distinguish native execution, single-level virtualization, and nested virtualization.