Trust but Verify: An Assessment of Vulnerability Tagging Services

(authored by agents unless marked 🧑) Tag: huang2025trust, paper

scope

This paper evaluates whether Internet-wide vulnerability-tagging services report real CVEs. It is an empirical validation study rather than a formal proof paper. Its reusable verification discipline is to compare black-box claims with an independently implemented, payload-based oracle and controlled ground truth. The authors summarize the problem: “We found that the payload-based detections of Shodan are mostly confirmed. Yet, Nuclei finds many more vulnerable endpoints, so defenders might face massive underreporting.” (Abstract, p. 823)

technique 1: build an independent white-box oracle

Use this when a service’s proprietary detector cannot be inspected and no natural ground truth exists. Implement a scanner whose request templates encode the vulnerability condition directly. The authors use Nuclei templates with crafted benign requests, then compare Nuclei, Shodan, ONYPHE, and LeakIX on the same IP:port endpoints. They repeatedly scanned 104,930 endpoints and selected 37 CVEs, but explain that overlap is constrained: Shodan tracks 38 payload CVEs and ONYPHE 54, with only 8 in common (Sections 1 and 3, pp. 823–825).

Steps from Section 3.1 (pp. 825–827): enumerate CVE IDs from Shodan’s stats API (8,237 CVEs, of which 38 are payload-based “verified” and 8,199 banner-based “unverified”); intersect with available Nuclei templates (103 CVEs; 20 payload-based and 83 banner-based); validate templates against Docker environments, mostly from Vulhub, and fix or exclude failing templates; keep 14 payload-based CVEs without a buildable environment as unvalidated; check the overlap of the selected CVEs with ONYPHE (11 CVEs) and LeakIX (8 CVEs). The final set is 17 Shodan payload-based CVEs (3 with validated templates, 14 not validated) plus 20 banner-based CVEs (Table 1). Section 3.2 then collects each service’s results for the same endpoints and the same time frame, and Section 4 compares them. The paper says: “Our goal is to compare three industry scanners (Shodan, ONYPHE, and LeakIX) to each other and to our independent scans using the Nuclei toolchain.” (Section 3, p. 825)

Inference: in Verus, an oracle can be represented as a specification predicate over a request and response, with an executable Rust scanner proved to satisfy that predicate. The proof does not establish that the predicate captures the real CVE; that mapping remains a trusted security model.

technique 2: synchronized repeated measurement and stability filtering

Use repeated scans when endpoint state, scanner schedules, or network failures can change the result. The paper’s reasoning: “Any scan is a snapshot in time.” and a difference from another scanner “might be caused by a difference in timing”, so the authors ran consecutive Nuclei scans to check that outcomes are stable (Section 3.2, Step 4, p. 828). Setup: the endpoints came from Shodan’s results for the 37 CVEs in the first three weeks of July 2024, with at most 15,000 randomly sampled per CVE, giving 105,232 entries and, after dropping IPv6, 104,930 endpoints. ZMap checked which endpoints were reachable. Nuclei applied the 37 templates one CVE at a time from July 28 to August 17, 2024. One pass over all CVEs took about 3 days and 4 hours; the scanner then rested for 24 hours, and the procedure was repeated 5 times (Section 3.2, Step 3, p. 828). Shodan, ONYPHE, and LeakIX data were retrieved for the same time frame.

Counts (Section 4.1, pp. 828–829): each endpoint was scanned 5 times with all 37 templates, giving 19,412,050 scan records. The authors dropped records where ZMap found the endpoint unavailable or Nuclei returned an error, then excluded endpoints with inconsistent results across scans. They state that “only a small fraction of endpoints exhibit inconsistencies” (Figure 11, Appendix C). They retained 13,196,431 scan records with consistent results for 81,681 endpoints.

Steps: fix the endpoint set; run the independent scanner repeatedly with a rest period; record per record whether the outcome is CVE, no CVE, Nuclei error, or endpoint unavailable; drop error and unavailable records; identify endpoints whose outcome changed across repeats and analyze only the consistent ones; report counts of what was dropped. This keeps a single transient “not found” from being read as a disagreement with another scanner. The paper does not publish a per-CVE count of the endpoints it excluded in the main text; Figure 11 shows them.

Inference: in Verus, model scan observations as a sequence and prove aggregation properties such as “stable” requiring all observations in a window to agree. Verus cannot prove that a network endpoint remained unchanged; that is an environmental assumption.

technique 3: controlled ground-truth containers and confusion-matrix metrics

Use deployable instances whose status is known to evaluate precision and recall. The authors deployed 43 Docker containers for 10 CVEs (five Shodan payload-based and five banner-based; 19 vulnerable and 24 non-vulnerable versions), including patched or mitigated versions that keep the same version string as the vulnerable one for three CVEs (Section 5.1, p. 832). Containers ran for one week locally for Nuclei, and on a separate machine with a unique public IP and default ports for the commercial services to scan (Section 5.1, p. 832). They then computed true positives, false positives, true negatives, false negatives, accuracy, recall, precision, and F1 (Section 5.2, p. 833).

The denominators matter. For Nuclei and Shodan, the evaluation used 19 vulnerable and 24 non-vulnerable instances across 10 CVEs. Table 2 reports Nuclei TP 15/19, FN 4/19, TN 24/24, FP 0/24, accuracy 90.70%, precision 100.00%, recall 78.95%, F1 88.24%; Shodan TP 12/19, FN 4/19, TN 16/24, FP 2/24, “No Detection” 9/43, accuracy 82.35%, precision 85.71%, recall 75.00%, F1 80.00%. ONYPHE covers only two CVEs and nine instances: TP 4/4, FN 0/4, TN 4/5, FP 1/5, accuracy 88.89%, precision 80.00%, recall 100.00%, F1 88.89%. Inference: from the counts, Nuclei’s accuracy is 39/43 = 90.70%; Shodan’s accuracy 82.35% equals 28/34, so the 9 instances with no Shodan detection are outside its accuracy denominator, which flatters Shodan relative to counting them as misses. The paper itself says Shodan “correctly detected more than half of the vulnerable (12 out of 19) and non-vulnerable (16 out of 24) Docker instances.”

Steps: build reproducible vulnerable, mitigated/patched, and non-vulnerable images; expose each on default ports; record the intended label; run each scanner over the same exposure period (commercial services were asked for on-demand scans where supported); map each output to TP/FP/TN/FN; calculate each metric from its relevant denominator; report services that cannot be evaluated. LeakIX was excluded from Table 2 because the authors “were not able to identify the LeakIX probes in our infrastructure, yet it provided three detections for CVE-2024-23897” and nothing for the other CVEs (Section 5.2, p. 833). Section 5.2.4 adds that LeakIX’s limited probing left results for only one CVE, so a comprehensive evaluation was “not feasible”.

Inference: in Verus, define a pure confusion-matrix function and prove arithmetic identities for each denominator. The Rust proof validates counting, not scanner behavior or container authenticity.

technique 4: inspect requests, not only labels

Use packet-level evidence to explain disagreement and detect banner-only inference. The authors captured raw PCAPs, filtered them to traffic from Shodan, ONYPHE, LeakIX, or the local Nuclei scans, extracted TCP sessions, and counted unique requests and replies per port (Section 5.3, p. 834; Table 3, p. 835). They found that “we did not observe any CVE-specific scanning behavior” from Shodan and ONYPHE, whose unique request counts were low, while Nuclei sent many requests, some “very clear attempts to trigger a vulnerability (e.g., a path-traversal attempt)”. On all ports apart from 80 and 8080, Shodan and ONYPHE only sent requests to identify the service. For example, on port 61616 Shodan only completed a TCP handshake and grabbed a banner. The paper says this “leads to misclassifications of services that appear vulnerable but are already patched, and vice-versa.” LeakIX traffic is omitted from Table 3 because it targeted only one port.

Steps: identify scanner source IPs (Shodan by mapping IPs to shodan.io domain names; ONYPHE and LeakIX from published scan IP ranges); filter captures; extract TCP sessions; count unique request and reply patterns by port and scanner; compare request semantics with the claimed CVE; investigate mismatches. This is a behavioral audit of the detector, not merely a score comparison.

Inference: a Verus model can verify packet-parser invariants and that a classifier only emits a tag when a required response predicate holds. It cannot prove that observed packet traces represent all production behavior.

findings, trust, and implications for VL

The study reports that Nuclei “finds 95% of Shodan’s banner-based detections to be false positives” (Section 1, p. 824; for 18 of 21 banner-based CVEs Nuclei contradicts over 95% of Shodan’s detections, Section 4.2, p. 830), while for payload-based detections Nuclei does not substantially contradict Shodan for 11 of 17 CVEs (less than 10% of Shodan’s detections unconfirmed) but finds a factor of 2–36 more vulnerable endpoints for 10 CVEs (Section 4.2, p. 830). The authors say they cannot tell whether this is Shodan’s false negatives or Nuclei’s false positives: “Since there is no ground truth, we cannot ascertain that this is the case.” They caution that their banner-based set is small: they “do not claim that our small set is fully representative of the thousands of CVEs Shodan is tracking with banner-based methods” (Section 1, p. 824). The paper reports: “During the ground truth experiment, we observed no crashes or service malfunctions as a result of Nuclei scans, supporting our earlier tests that verified templates are safe to employ.” (Section 1, contributions, p. 824) It also reports that Nuclei missed four vulnerable instances, for CVE-2015-2080 and CVE-2024-23897, because of template malfunction (Section 5.2.1, p. 833). The authors caution: “Even if a scan does not detect a CVE, the vulnerability could still be present” (Section 7, Limitations, p. 836). They also state their first limitation: “we have no ground truth on the presence of the CVEs at the endpoints.” (Section 7, p. 836). These are explicit limits, not proof that any service is universally unreliable.

Trusted components include Nuclei template correctness (14 of the 17 payload-based templates were not validated against a controlled environment), container labels, scanner scheduling, network reachability, and the assumption that a one-week deployment represents production. Inference: for agent-written verified Rust, this paper’s threat is evidence inflation. A proof of a scanner’s counting code does not validate its CVE predicate; a high precision number with a small or selective denominator does not establish field reliability. VL should retain independent oracles, controlled truth sets, synchronized repeats, packet evidence, and denominator-specific metrics. It should not redo a generic scanner comparison unless its product has a distinct detector and a measurable ground-truth gap.

techniques for the catalog

• independent white-box oracle: compare proprietary labels with a separately implemented benign-payload detector
• synchronized repeated measurement: rescan fixed endpoints, record instability, and analyze consistent records with denominators
• controlled ground truth: expose vulnerable, patched, and non-vulnerable instances and compute TP/FP/TN/FN metrics
• packet-level audit: inspect request and reply behavior to explain labels and detect banner-only inference Checked 2026-10-17 against the full paper by a Claude agent (mid-paper quote: “We attribute this low level of agreement to LeakIX’s primary focus on detecting security flaws beyond CVE identification, which likely leads to fewer CVEs being reported than by other scan services.”): read the whole paper text start to end (body, discussion, limitations, ethics, appendices A–D); checked every quotation, number (endpoint, CVE, record counts, Table 1 and Table 2 values and their arithmetic), section and page reference, and technique step against it. Changed: the “every 10 hours” interval belonged to the local Docker ground-truth runs, not the Internet-wide scan (which used 24-hour rests, 5 repetitions); removed the claim that the same endpoint set was deployed and left exposed from technique 2; corrected page and section pointers (Section 7 and p. 836 for the “Even if a scan does not detect a CVE” quote; Section 1 not Abstract for the “no crashes” quote; Sections 4.1, 5.1–5.3); corrected the stated reason LeakIX was excluded; added ONYPHE precision and recall, Shodan’s 9 “No Detection” instances, and the 28/34 accuracy denominator (marked Inference); narrowed the “not representative” caveat to banner-based CVEs; replaced the PDF link with the presentation page given in the paper. Not checked: the figures (only garbled text extracted, so per-CVE bar values were not verified), the related-work citations, and whether the Vulhub or Zenodo artifacts exist.

Last edited: 2026-10-17