Benchmark

This page carries two results. The latest one, measured on vulkro 0.30.0 against blind holdout applications, comes first. The earlier Tier 1 corpus run on vulkro 0.28.0 follows it, unchanged, with its own method. The two are different tests on different code and are never added together.

Latest result: vulkro 0.30.0 on blind holdouts

Nine blind applications that nobody working on the engine reads: three in Go, one in Java, two in JavaScript / TypeScript and three in Python. Each carries planted vulnerabilities with an expected severity, known-safe controls that look like vulnerabilities but are not, and a list of the HTTP endpoints it exposes.

	vulkro 0.29.0	vulkro 0.30.0
Planted vulnerabilities found	122 of 229 (53%)	181 of 229 (79%)
Missed	107	48
Real Criticals rated Critical	6 of 90 (7%)	32 of 90 (36%)
False Criticals (Critical on safe or unlabelled code)	9	20

On vulkro 0.30.0 the scan also flagged 39 of 152 known-safe controls (26%) at Medium or above, and mapped 378 of 390 expected endpoints (97%) with the right path and method.

Read the last row of the comparison as plainly as the first: false Criticals went up, from 9 to 20. Finding issues improved faster than rating them, and that is the gap the next release has to close.

By application

All-confidence view. Each row is one blind application; the last row is the sum of the nine.

Application	Found	Missed	Criticals rated Critical	False Criticals	Safe code flagged	Endpoints mapped
Go, app 1	16/27 (59%)	11	4/10	4	5/18	40/40
Go, app 2	16/21 (76%)	5	2/9	0	2/16	29/29
Go, app 3	20/23 (87%)	3	5/9	1	4/17	37/38
Java, app 1	28/31 (90%)	3	8/10	0	2/14	38/39
JS / TS, app 1	16/26 (62%)	10	1/11	5	5/19	53/55
JS / TS, app 2	26/29 (90%)	3	1/11	1	4/15	41/45
Python, app 1	22/26 (85%)	4	4/11	5	5/19	52/53
Python, app 2	17/22 (77%)	5	3/9	3	4/18	50/50
Python, app 3	20/24 (83%)	4	4/10	1	8/16	38/41
All nine	181/229 (79%)	48	32/90 (36%)	20	39/152 (26%)	378/390 (97%)

Provenance: vulkro 0.30.0, measured 2026-10-05, nine blind holdout applications kept unread by the engine team, scanned offline and deterministically.

Method

• Blind to the engine team. Nobody working on the engine reads a holdout application, its labels or its per-issue results. The scorer runs in redacted mode and writes only aggregate and per-class numbers: no entry ids, files, lines or issue text. Fixes are motivated and validated on separate development applications.
• Planted issues and negative controls. Every application carries planted vulnerabilities, each with an expected severity, and known-safe controls. The controls are what catch a scanner that flags everything.
• Offline and deterministic. Each scan runs from a pinned binary with the network off (VULKRO_OFFLINE=1), the result cache off and deterministic mode on. The same binary on the same applications gives the same counts.
• Matching. Each finding is attributed to the nearest labelled entry in the same file within eight lines. A planted issue counts as found when a finding at Low confidence or above is attributed to it. Info findings never count.
• Severity. A real Critical counts as rated Critical only when a finding attributed to it is itself Critical. A false Critical is a Critical finding attributed to a known-safe control or to no labelled entry.
• Safe code flagged. A known-safe control counts as flagged when a finding at Medium or above is attributed to it.
• Endpoints. An expected route counts as mapped when the scan reports the same path, after parameter syntax is normalised, with a compatible HTTP method.
• Confidence view. These figures are the all-confidence view (vulkro scan . --all-confidence). The default view hides low-confidence findings, so on its own it shows fewer of these.

What this number is not

• It is not a comparison. No other scanner was run on these applications. The result measures Vulkro against its own previous release on the same code.
• It is not real-world precision. Applications built to carry planted issues are denser than production code. How often a finding on a real production codebase is a real issue is a separate, harder measure. We track it and are still improving it, and it is not what this table reports.
• It is small. Nine applications, with one in Java and two in JavaScript / TypeScript. A per-language figure from one or two applications shows a direction, not a settled rate.
• The holdouts are private. Publishing a blind application would end its use as a blind test, so the applications and their labels are not released. The method above is.

Earlier run: vulkro 0.28.0, Tier 1 corpus

This is the result this page used to lead with, kept as the earlier run. It is a different test (public, deliberately vulnerable repositories with documented bugs, scored at --min-confidence high against other scanners) and its figures are not comparable with the holdout result above. The roll-up comes from the vulkro 0.28.0 scorecard of 2026-09-18; some sections below (the Java note, the speed table and the tool-version table) still describe the earlier vulkro 0.15.0 run they were generated from, and say so.

Vulkro is measured against six other scanners on one ground-truth corpus, on the same machine, on the same day, with the same scoring code. Every bug in the corpus is pinned to published provenance in a real project (the provenance rule below): a CVE or GHSA advisory where one exists, and otherwise the upstream project's own in-source vulnerability marker or documented challenge, which is how deliberately-vulnerable corpora such as Juice Shop, NodeGoat and DVGA identify their planted bugs. A finding counts only within five lines of the catalogued bug. Every table on this page, including the comparisons we lose, is scored that way. The methodology is published in full; you can check any number against the advisory it cites.

Headline results

The four tools that scan more than one language, rolled up over the scored corpus: 83 catalogued bugs (one repo whose primary language Vulkro does not yet support is excluded from the roll-up; the scorecard states the exclusion). Vulkro is scored at --min-confidence high, the recommended CI setting; the other tools' findings are never confidence-filtered (see Methodology for why).

Read the precision row as exactly this and nothing more: precision against the catalogued-bug set of this curated corpus, at high confidence. It is not a claim about the share of all findings that are real on an arbitrary repo; the "what this precision is measured over" note below the table states the difference explicitly, and every figure regenerates from the committed scorecard.

	Vulkro	Tool A 2.0.2	Tool B 1.136.0	Tool E 1.25.0
Precision	0.62	0.37	0.53	0.53
Recall	0.40	0.52	0.20	0.20
F1	0.49	0.43	0.30	0.30
Catalogued bugs found (of 83)	33	43	17	17
False positives	20	72	15	15
TP / FP / FN	33 / 20 / 50	43 / 72 / 40	17 / 15 / 66	17 / 15 / 66

Why the other tools are not named. They appear as Tool A through Tool F, here and in every table below. That is site policy: we do not put other vendors' names on our own pages. The labels are stable across this page, and each tool keeps the pinned version number it ran at. State the cost plainly: with the names withheld you cannot take a row from this page and go check it against a named product yourself. What is not withheld is everything that decides what the numbers mean. The corpus and its provenance rule, the matching rule, the four buckets, the scoring formulas and the exact commands the harness runs are all published below, and the rows Vulkro loses are printed at the same size as the rows it wins. We do not publish the mapping, and the order of the labels carries no information.

How to read it:

• Precision first. When Vulkro fires at high confidence, it is right 0.62 of the time. That is the highest precision of any multi-language tool in the run.
• Against the free baseline. Vulkro finds 1.9x the catalogued bugs Tool B finds (33 vs 17) at higher precision (0.62 vs 0.53). Tool B and Tool E score identically because both lanes run the same live community security ruleset; the point of the Tool E lane is comparing engines on identical rules.
• Where a rival scores higher. Tool A finds more of the catalogued bugs (43 vs 33) and pays for it with 72 false positives to Vulkro's 20, and about eleven times the runtime. Higher recall, far lower precision: the trade this whole page is about.
• What this precision is measured over. These figures count a finding against the catalogued-bug set. Vulkro emitted 300 total findings in this run, so the share of ALL findings that map to a catalogued bug is far lower than 0.62. That is expected for a scanner that also reports IaC, secrets and dependency issues the corpus does not catalogue, but it is not the same statistic and we do not present it as one.
• The PHP and Java repos are not hidden. All four tools also scan DVWA (PHP), WebGoat, and java-sec-code (Java); the per-repo rows are published in the scorecard. They are excluded from these totals for two different reasons. PHP is out of Vulkro's language scope. Java is now analysed - see Java and Spring - but that support shipped after this corpus was scored (these numbers come from a Vulkro 0.15.0 run), so the Java repos stay out of the roll-up pending a re-score. Tool A's wins on those repos are called out in Where Vulkro misses.

The corpus, version 2026-07-05

The Tier 1 corpus is 15 public repositories, pinned by SHA: 12 deliberately-vulnerable apps with 90 catalogued bugs, plus 3 clean framework controls (express, fastapi, flask) that must produce zero findings. 76 of the 90 bugs are in the supported-language roll-up above; the other 14 live in the PHP and Java repos.

We grew our own ground truth from 69 to 90 bugs in July 2026 and published the recall drop that came with it. The growth batch added two repos chosen to be hard for us (DVGA, a Python GraphQL app, and govwa, a Go app - both score 0 Vulkro recall today) and 21 previously-missed bugs in existing repos. The corpus is versioned; every change is in the changelog.

The provenance rule. Every catalogued bug must be independently verifiable and must cite the repo's own vulnerability documentation as the primary source: its README or solutions doc, an in-source marker (juice-shop's vuln-code-snippet comments, NodeGoat's tutorial pages), or a published CVE / GHSA pinned to the commit. Each entry is then confirmed by reading the sink at the pinned SHA. Selection is not biased toward what Vulkro detects: catalogued bugs Vulkro misses stay in the corpus as legitimate false negatives. That honesty is the point of the benchmark.

[[vuln]]
file = "app/routes/contributions.js"
line = 32
class = "command-injection"
cwe = "CWE-94"
note = "eval(req.body.preTax) - A1 SSJS injection (source comment line 31)"

Methodology

Ground truth and matching

Each catalogued bug is a (file, line, class) tuple. A finding matches a ground-truth entry iff the files are equal and the finding line is within 5 lines of the catalogued line (+/-5). Each ground-truth entry can be claimed by at most one finding (greedy nearest), so over-firing on the same bug does not inflate the score.

The four buckets

bucket	meaning
TP	Tool fired on a catalogued bug (same file, line within +/-5).
FP	Tool fired in a file we catalogued, but at no catalogued bug line.
FN	Catalogued bug, no tool finding fired against it.
OOS	Tool fired in a file we did not catalogue. Neither rewarded nor penalized: we do not claim to know whether those files are clean or just unannotated.

The OOS bucket is what separates "noisy" from "broad": a tool that fires thousands of times outside the catalogued files is neither punished for it nor credited with it.

Confidence filtering: Vulkro only, and why

Vulkro is scored at --min-confidence high for the headline (its recommended CI setting) and at low for the labeled max-recall run. The other tools' findings are never confidence-filtered, because their confidence ladders are not directly comparable to ours: dropping another tool's "low confidence" findings using our idea of what "low" means would silently rig the comparison. Every other lane runs at its own defaults instead.

Metrics

• precision = TP / (TP + FP): when the tool fires, how often is it right?
• recall = TP / (TP + FN): of the catalogued bugs, how many did it find?
• F1 = 2 x (precision x recall) / (precision + recall): the standard combined score.

Per-language specialists: the fair-fight rule

Tool F only scans Python, Tool D only Go, Tool C only JavaScript. Rolling a single-language tool up over a mixed corpus would manufacture zeros for it, so the specialists never enter the roll-up. Instead, each specialist is scored only over the repos matching its language, and Vulkro is re-scored over the same repo subset. Every comparison below is a fair fight. Vulkro wins two of the three; the one it loses is printed just as large.

Tool F vs Vulkro, 7 Python repos (42 catalogued bugs)

	Tool F 1.9.4	Vulkro (same repos)
Precision	0.56	0.87
Recall	0.43	0.64
TP / FP / FN	18 / 14 / 24	27 / 4 / 15

On its home language, Tool F finds 18 of the 42 catalogued Python bugs with 14 false positives; Vulkro finds 27 with 4.

Tool D vs Vulkro, 1 Go repo (4 catalogued bugs): Tool D wins

	Tool D 2.27.1	Vulkro (same repo)
Precision	0.20	0.00
Recall	0.75	0.00
TP / FP / FN	3 / 12 / 1	0 / 1 / 4

On our one Go repo, Tool D finds 3 of 4 bugs; Vulkro finds 0. Go detection depth is a known gap we publish. The four misses (a fmt.Sprintf SQL interpolation, an html/template XSS, and two HttpOnly: false insecure cookies in govwa) stay catalogued as false negatives rather than being quietly dropped from the corpus. If Go is your primary language, Tool D is free and it beats us there today.

Tool C vs Vulkro, 4 JS/TS repos (30 catalogued bugs)

	Tool C 0.4.3	Vulkro (same repos)
Precision	0.50	0.77
Recall	0.33	0.57
TP / FP / FN	10 / 10 / 20	17 / 5 / 13

One caveat that the per-repo rows keep visible: Tool C targets plain JavaScript, so on the TypeScript repo (juice-shop) it sees almost nothing in .ts files and scores 0 of 8 catalogued bugs there.

Max-recall mode

Drop the confidence threshold (--min-confidence low, the default) and Vulkro trades precision for recall. The published scorecard labels this run as max-recall:

metric	Vulkro max-recall	Vulkro headline (high)
Precision	0.15	0.62
Recall	0.76	0.40
TP / FP / FN	63 / 347 / 20	33 / 20 / 50

Max-recall finds 63 of the 83 roll-up bugs at the cost of 347 false positives. It is the right setting for a one-shot audit where a human will triage everything once; --min-confidence high is the right setting for CI. We publish both so the trade-off is yours to make, not ours to hide.

Where Vulkro misses

The corpus is built to keep our failures on the record:

• GraphQL resolver taint. DVGA (Damn Vulnerable GraphQL Application) has 6 catalogued bugs reached through GraphQL resolvers. Vulkro finds 0 of them. Tracing taint through resolver dispatch is an open detection gap.
• Go. govwa has 4 catalogued bugs; Vulkro finds 0, Tool D finds 3 (see the fair-fight table above).
• PHP and Java. DVWA (7 bugs, PHP) is outside Vulkro's language scope entirely. WebGoat (4) and java-sec-code (3) are Java: this scorecard was generated with Vulkro 0.15.0, before Java/Spring support shipped, so those repos score 0 here and stay excluded from the roll-up until the corpus is re-scored with Java analysis on. Tool A finds 9 of those 14 catalogued bugs. Java coverage today is documented on the Java and Spring page.

By bug class, the 32 headline-run misses across the supported-language repos concentrate in: SQL injection (11), command injection (8), XSS (6), open redirect (4), plus single-digit counts of insecure deserialization, insecure cookies, SSRF, JWT verification, path traversal, and LFI. The full FN-by-class table is regenerated in every scorecard.

Speed

Wall clock for the whole 15-repo corpus, one full end-to-end process invocation per tool per repo. Measured 2026-07-04 on an Apple M5 Pro, 24 GB RAM, macOS 26.5.1.

tool	whole-corpus scan time
Tool D 2.27.1	0.6s (1 Go repo only)
Tool F 1.9.4	3.2s (7 Python repos only)
Tool C 0.4.3	24.1s (4 JS/TS repos only)
Vulkro 0.15.0 (high confidence)	38.0s
Vulkro 0.15.0 (default)	39.3s
Tool E 1.25.0	46.1s
Tool B 1.136.0	48.9s
Tool A 2.0.2	363.0s

The honesty notes that come with that table:

• This reference is a single cold run per tool per repo (--timing-runs 1), because the 7-tool matrix makes median-of-3 a very long session. For publishable single-tool comparisons the harness supports --timing-runs 3 and we recommend it; the aggregation line embedded in the scorecard records exactly what each run did.
• Repos are pre-cloned to local disk before timing; clone time is excluded. Every sample includes process start, scan, and report serialization.
• Tool-internal result caches are disabled where the tool supports it (Tool A runs with its cache forced off), so every sample is a cold scan. Tool A's incremental cache makes its repeat scans on unchanged code much faster than shown here; the table just does not mix cold and warm numbers.
• The specialists only scan their matching-language repos, so their totals cover only that subset. A 0.6s single-language total is not comparable to a 38.0s whole-corpus total and the table says so.
• Wall clock is machine-dependent. Compare the ratios between tools on the same machine, not absolute seconds across machines.

How the numbers are produced

You do not have to take any number on this page on faith. Two things make it checkable. First, every catalogued bug cites a public CVE or GHSA advisory in a real project (the provenance rule above), so you can verify any single entry against its source. Second, the exact method is written down: the scoring rules, the five-line match window, the confidence setting each column is scored at, and the pinned tool versions in the table below.

Each configuration writes raw findings, normalized JSON, the markdown scorecard, runtime.json, and a timing record (hardware, run count, cache state, and the tool versions actually used), so a result is always traceable to the run that produced it.

One honest limitation: the corpus and the internal harness that runs it live in our private repository, so you cannot rerun our exact scoring yourself. We publish the method, the per-bug advisories, and the comparisons we lose, not a downloadable corpus. The check that decides anything is the scanner run on your own code, at the same --min-confidence high setting the headline row is scored at.

What "reproducible" means for the run itself: same Vulkro binary + same corpus SHAs + same scoring code = same TP/FP/FN counts, every run, on any machine. Known sources of drift:

• The shared community ruleset is not pinned. The ruleset behind the Tool B and Tool E lanes is served live by its own registry and changes over time, so those two counts can move without any change on our side. Our corpus SHAs and scoring code are pinned; someone else's rule feed is not ours to pin.
• Other tools' versions. Newer releases change rules. The scorecard records the versions it ran; compare like with like.
• Hardware moves speed numbers. Compare ratios, not absolute seconds.

Tool versions in the published run

tool	version	note
Vulkro	0.15.0	scored at --min-confidence high (headline) and low (max-recall)
Tool A	2.0.2	SAST lane only, result cache disabled
Tool B	1.136.0	community security ruleset, fetched live, not pinned upstream
Tool C	0.4.3	JS/TS repos only; bundles a third-party pattern engine (1.86.0)
Tool D	2.27.1	Go repos only (specialists section)
Tool E	1.25.0	same live community security ruleset as the Tool B lane
Tool F	1.9.4	Python repos only (specialists section)

Tools not included

One authenticated commercial scanner is deliberately left out of the published runs. It cannot run without an authenticated account, and publishing benchmark results may be restricted by its terms of service (a check we have not completed). The scorecards carry a standing "Tools not included" note so this gap is a documented decision, not a silent one.

Two DAST products are also out. They cannot scan a static code directory at all, so a static-corpus comparison would not be meaningful for them.

Caveats: what these numbers do not tell you

• The corpus is deliberately-vulnerable code. Real-world performance varies; deliberately-vulnerable apps are denser in bugs and lighter in framework noise than production code. The 3 clean framework controls (express, fastapi, flask) are the noise-floor check inside the corpus.
• Class matching matters. A finding under the wrong class does not count as a TP even when the line is right. This penalizes every tool roughly equally.
• Ground truth is a floor, not a census. The 90 catalogued bugs are the bugs we could verify by provenance; the OOS bucket exists precisely because we do not pretend the rest of every file is clean.
• A benchmark is one input. Run the tools on your own code. The harness exists so that "run it yourself" is a one-command suggestion rather than a slogan.

Related

• Confidence model - how High vs Medium vs Low is computed from evidence weights.
• OWASP API Top 10 - per-class detector inventory.
• Reproducing in CI - running the benchmark as part of every PR.