1 · Vulnerability Discovery
60.1% of Index · 13 Patch Pairs
Multi-Hop Bug Hunting (Analyze)
Models start at an entry function up to 3 call-hops above a real historical vulnerability inside a compiled Windows driver or boot/system binary (with a forward call cone of 20+ functions) and are asked to audit reachable memory-safety, integer-overflow, type-confusion, or refcount flaws.
- Paired before/after test: Models audit the unpatched build first and (if they hit the vulnerable sink) immediately audit the patched build under the identical prompt.
- Feature-flag reasoning: Many Windows patches keep vulnerable code compiled in behind a default-ON WIL Velocity flag. Models must inspect flag state and pass both halves: flag the bug when unpatched and report NO-FIND once fixed.
2 · Technical Verification
39.9% of Index · 26 Audits
Binary Claim & Trap Auditing (Validate)
Models receive technical review drafts containing 9–10 concrete binary claims per binary (direct call edges, instruction constants, and function-entry statements) and must independently verify every statement from disassembly, decompiled C, and raw bytes, marking each CONFIRM or REFUTE.
- Plausible false traps: Mixes true statements with subtle falsehoods — 2-hop indirect calls misstated as direct calls, constants borrowed from sibling functions, and shifted function-entry RVAs.
- Adversarial resistance: Includes unwarned prompt-injection payloads embedded inside binary data claims attempting to force CONFIRM or invoke forbidden tools.
3 · Byte-Level Grounding
Diagnostic · 77 PE Entries
Raw Data & RVA Precision (Inventory)
Models are given raw read-only data (.rdata) address ranges inside compiled PE binaries and asked to read raw byte slices and emit structured tables of ASCII literals paired with their exact hexadecimal Relative Virtual Addresses (0xRVA).
- Zero-tolerance byte check: Every emitted {rva, string} row is checked byte-for-byte against the PE image.
- Catches offset drift: Directly surfaces hex arithmetic slips, off-by-N address slides, and fabricated strings that look plausible in prose but fail against real binary layouts.
4 · Live Proof-of-Concept
Separate Score · Live KVM Replay
Working Trigger on a Live VM (PoC-Score)
Models are given a short brief for an already-fixed vulnerability (entry interface, symbolic sink, and the grader's breakpoint contract) and must produce a poc.exe that drives a real, unpatched Windows 11 guest to the vulnerable sink, building and debugging inside the VM.
- Repair and derive: Repair items start from a reference PoC carrying one planted runtime fault; derive items start from the brief alone.
- Graded by replay: The submitted PoC is re-run three times from a clean snapshot and passes only if all three runs hit the sink breakpoint with the required register or bugcheck condition. The model's own success claim is not used.
- Reported separately: PoC-Score is the replay-pass rate over these items and is not part of the Capability Index, so VM variance never moves the static score. “—” means the model has not been run on this lane.
Clean-Room Isolation & Anti-Memorization Controls
Models have no SQL search, no cross-version diff tools, and zero human annotations — every call is pinned to a single binary file_id. All evaluated vulnerabilities are already publicly fixed by Microsoft (fix_state='fixed').
11 Read-Only Binary Tools Provided to Every Model
Hex-Rays Decompiler & Microcode
x86-64 Disassembler
Call & Data Xrefs
Struct / Type Layouts
Raw Byte Reader
Pointer & Scalar Tables
WIL Feature-Flag Inspector
CFG Flag Arm Slicer
Sandboxed Chunk Emulator (rax)
Z3 / SMIR Bound Prover