◊ Methods

Methodology

How the measurement works — sampling, capture, enrichment, analysis — and, just as importantly, where it is biased and what it cannot tell you.

Research question

Across a uniform random sample of the reachable public-IPv4 web, where do CVE-associated and reputation-flagged services concentrate — by network (ASN / organization), geography, product and version, and port — and how does that exposure change over time?

The observatory is organized around measuring that concentration, rather than building an exhaustive index of every host. It samples the ordinary, reachable web, keeps the latest record for each observed ip:port service, and stores daily aggregate snapshots for longitudinal analysis.

Sampling

Scanning agents generate uniform random public IPv4 addresses and probe a small, fixed set of common web ports (80, 443, 8000, 8080, 8443) with nmap. Reserved, private, and special-use ranges are excluded, and every candidate address is checked against an operator exclusion list before it is touched. Discovery runs continuously at a deliberately slow rate; this is a random sample over time, not a synchronized exhaustive sweep.

The sampling unit is one IPv4 address. Go's standard math/rand/v2 generator draws uniformly from all 32-bit values; candidates in the maintained special-use, private, documentation, multicast, loopback, and operator-exclusion CIDRs are rejected and redrawn. Addresses are unique within each batch of 10 but can be drawn again in later batches. Conditional on not being excluded, every eligible address has the same probability of being selected on each draw. The current agent runs at most two host workers concurrently.

Capture

For any address answering HTTP or HTTPS on those ports, an agent records what an anonymous visitor would see: a screenshot taken in a headless Chromium browser, the service banner, HTTP status, the TLS certificate common name, coarse IP-based geolocation, and structural hashes of the page (a perceptual hash of the screenshot and a hash of the DOM structure). Each record is a point-in-time snapshot keyed deterministically by ip:port, so re-observations update the same record rather than duplicating it.

Discovery runs nmap -sV -n -T3 by default and is bounded to five minutes per 10-address batch. A responding service is loaded in Chromium at a 1280×720 viewport with a 15-second navigation budget, followed by a configurable settling delay (currently two seconds) before DOM text and a screenshot are captured. Ports 443 and 8443 start with HTTPS; other ports start with HTTP and retry once with HTTPS only after an SSL protocol error. Redirects follow Chromium's normal behavior, and the recorded secured value reflects the final URL. Certificate errors are ignored so publicly reachable misconfigured TLS services remain observable. Captured page text is truncated to 32,760 bytes.

The agent does not consult page-level robots directives before the initial public-page load: those directives govern automated content indexing, while this project performs a measurement of service reachability. It does honor the project exclusion list before any network contact and publishes a permanent opt-out process.

The agents do not sign in, submit credentials, exploit or fuzz anything. Network discovery checks only the five listed web ports; when a service responds, the agent loads the page a browser would and nothing more. Every HTTP page fetch carries a self-identifying User-Agent linking to the scan-info page, so operators can see who we are and opt out.

Enrichment

When enrichment is enabled and the provider is reachable, eligible captured hosts are cross-referenced server-side against public security data. Shodan's keyless InternetDB can supply associated CVEs, tags, and other open ports; on the record view, additional configured feeds may be queried (VirusTotal, AbuseIPDB, GreyNoise, AlienVault OTX, ThreatFox, IPQualityScore, Pulsedive, IPinfo, ip-api, and RIPEstat). Missing credentials, provider errors, rate limits, or disabled enrichment can produce partial or absent results. Returned records preserve contributing sources and a last-enriched timestamp where available; these are provider associations, not independently verified findings.

A CVE association means a third party has linked the host's software or network to a known vulnerability identifier — it is not proof the host is exploitable or unpatched. Reputation assessments can be wrong or stale. CVEs, CPEs, hostnames, and additional ports from InternetDB describe the host, not necessarily the particular web service shown on the record. The observatory reports all of these as third-party signals and never as ground truth.

Analysis

The study aggregates these observations to ask where exposure clusters: the share of cleartext vs. TLS services, the distribution of CVE-associated hosts across ASNs and organizations, geographic concentration, which products and versions recur among flagged hosts, and how those distributions drift over time. The live statistics view is a window into these aggregates. Researchers can save a timestamped live export for reproducible offline analysis; first-party dated archival exports are planned but are not yet published.

Operational definitions

Candidate address: an eligible IPv4 address returned by the rejection sampler.
Reachable service: one of the five configured ports that nmap reports open and whose page Chromium captures successfully.
Host: one distinct IPv4 address; a host can contribute multiple service records.
Service: the latest retained observation for one ip:port.
Cleartext: the final captured page URL used HTTP rather than HTTPS.
CVE-associated: InternetDB returned at least one host-level CVE identifier.
Provider-flagged: a configured reputation provider contributed evidence summarized as suspicious or malicious.

Dashboard percentages are descriptive ratios within the selected retained-service window: count meeting the displayed condition divided by the displayed service denominator. Concentration percentages require at least 20 services in a displayed group. They are not presently weighted population estimates and do not include confidence intervals.

Limitations & biases

Reading the data honestly means keeping these in view:

Single vantage point. Observations come from one network egress. Hosts that block that network, or that respond differently by source, are under- or mis-represented.
Sampled, not exhaustive. Unlike exhaustive scanners (Rapid7 Project Sonar, Censys), this is a random sample. It estimates distributions, not complete inventories, and rare phenomena may be missed.
Survivorship & scope. Only hosts answering HTTP/HTTPS on five common ports are captured — a deliberate, narrow slice of the internet, not "the internet."
Point-in-time. A record reflects one moment; the host may have changed, moved, or gone away since. Re-observation replaces the prior service record; individual capture history is not retained, while daily aggregate snapshots preserve sample-level trends.
Third-party enrichment. CVE and reputation data inherit the coverage, latency, and error rates of the upstream feeds.
Coarse geolocation. IP-based geo is approximate and can misattribute hosts, especially on cloud and CDN networks.

Reproducibility

The collection and analysis code is publicly viewable in the repository; consult its provenance and licensing notices before reuse. The live record schema and export interface are documented on the data page. Dated archival snapshots are planned but are not yet published. The scanning conduct follows established norms for internet measurement — see ethics.