Part of the benchmark canon: 9 entries, all measure the model. How to read the Paladin column is defined on the canon page.
| Benchmark | Organization | Measures | Paladin |
|---|---|---|---|
| Cybench | Stanford and academic consortium | CTF tasks drawn from four professional competitions | Measures the model |
| Cybergym | UC Berkeley | Reproducing real vulnerabilities in open-source projects | Measures the model |
| CVEBench | Academic | Finding and exploiting known CVEs in web applications | Measures the model |
| 3CB (top 12) | Apollo Research / Apart Research | Offensive tasks mapped to MITRE ATT&CK | Measures the model |
| SOLVE | Irregular (Pattern Labs) | Difficulty scoring for vulnerability and exploit benchmarks | Measures the model |
| CyScenarioBench | Irregular (Pattern Labs) | Scenario-based cyber capability tests | Measures the model |
| NYU CTF Bench | NYU | Two hundred plus CSAW CTF challenges | Measures the model |
| HonestCyberEval / AIxCC | Alan Turing Institute / DARPA | Automated vulnerability discovery on real codebases | Measures the model |
| XBOW internal benchmark | XBOW | Miss rate on known CVEs in version-frozen applications | Measures the model |
Previous: Agent security and prompt injection. Next: Cyber defense and threat intel.