Architecture

The shape of the system

The observatory is a small distributed system with three moving parts: a scanner agent that discovers and captures hosts, a collector that ingests, stores, enriches, and serves the data, and a React frontend that is compiled and embedded inside the collector's binary so the whole product ships as one image. It is a Go + React monorepo backed by MongoDB and object storage, deployed on ordinary AWS primitives.

Scanning currently runs from one low-rate host on an ordinary connection. Moving it to a dedicated VPS is planned; until that happens, this page does not claim dedicated scanner infrastructure or publish a scanner IP. The agent is packaged separately so it can ultimately isolate browser rendering and scan traffic from the public service and database.

End-to-end data flow

• Discover. The agent generates uniform-random public IPv4 addresses and checks five common web ports with nmap, skipping anything on the exclusion list.
• Capture. For hosts answering HTTP/HTTPS it drives a headless Chromium (via chromedp) to screenshot the page and record the banner, status, TLS certificate name, and structural hashes.
• Submit. Results are packed into an HMAC-SHA256-signed, gzip-compressed envelope and POSTed to the collector — the same wire format used by the private Python agents behind What's on HTTP, created by elixx.
• Ingest. The collector verifies the signature, decodes, computes a deterministic ID per ip:port, uploads the screenshot to object storage, and upserts the record into MongoDB (buffering to disk if the database is momentarily unavailable).
• Enrich. A background worker cross-references each host against public CVE and reputation feeds, denormalizing the summary back onto the record.
• Serve. Clean JSON APIs power the embedded React UI, which renders the Overview, observation feed, search, statistics, and record pages.

The collector (Go)

The collector is a single statically-linked Go 1.26 binary that serves everything — the ingest endpoint, the read APIs, and the embedded UI — from one process on one origin (so the browser makes no cross-origin requests in production). The read API still sends a permissive Access-Control-Allow-Origin: * header for the separate Vite development server and read-only external research clients. Routing uses Go's standard-library method-pattern mux; there is no web framework.

Operational properties of the collector:

• Designed for failure. If MongoDB is unreachable, accepted submissions are spooled to disk as BSON and flushed when it recovers — ingest never drops data because the database blinked.
• Idempotent by construction. Each service's _id is derived deterministically from ip:port, so re-observing a host updates one document instead of duplicating it — essential for a continuously updated sample.
• Bounded concurrency & rate limits. Object-storage uploads run with a bounded worker pool; the public read APIs are throttled per client IP with an in-process token bucket.
• Strangler migration. The Go collector reimplements the exact legacy v1 wire protocol (HMAC + gzip + base64) of the private What's on HTTP Python system created by elixx. Representative outputs are verified byte-for-byte with golden tests, so old and new agents could run side-by-side during the rewrite rather than a risky flag-day cutover. The complete contribution history and current source-rights status are recorded in PROVENANCE.md.

Supporting packages handle the media work — perceptual (dHash) and DOM-structure hashing, JPEG thumbnail generation via golang.org/x/image/draw, and product extraction from banners — plus GeoIP lookups against a MaxMind GeoLite2 database.

The scanner agent (Go)

The agent is a separate Go program: random IPv4 batches → nmap discovery → optional headless Chromium capture → signed submit, with an RDAP lookup for network ownership. It runs deliberately slowly, honors the CIDR exclusion list it fetches from the collector, and sends a self-identifying User-Agent so operators can see who is visiting and how to opt out. It ships as its own container image (Debian + nmap + Chromium + fonts), because the browser and scanning tooling have heavier system dependencies than the collector.

The frontend (React + TypeScript)

• React 19 + TypeScript, built with Vite 8; client-side routing with React Router. A single typed API client wraps the collector's JSON endpoints and surfaces a clear collector unreachable state instead of a false no results.
• Bespoke SVG visualizations. The time series and bar charts are hand-built SVG tuned to the design system; the world map projects TopoJSON with d3-geo. This keeps the bundle lean and the charts exactly on-theme.
• Prerendered public routes. The production build renders complete HTML for public routes and then hydrates the same React application in the browser. Titles, descriptions, canonical/social metadata, a sitemap, a web manifest, and Dataset JSON-LD are available without executing JavaScript.
• Embedded in production. Prerendering happens at build time only; the generated dist/ is compiled into the Go binary with go:embed. Production needs no Node process, separate static host, second deploy, or CORS configuration.

Storage & the data model

Structured records live in MongoDB (one document per ip:port service, plus separate collections for enrichment cache, the CIDR blacklist, and daily rollups). Screenshots — the bulk of the bytes — live in object storage (Cloudflare R2 or Amazon S3 served through CloudFront), referenced from the record by key, with a base64-in-MongoDB fallback when object storage is disabled. A small server-generated JPEG thumbnail is stored alongside each capture so the card grid loads roughly a tenth of the bytes of the full screenshot.

The enrichment pipeline

When enrichment is enabled and providers are reachable, eligible hosts are cross-referenced server-side against Shodan's keyless InternetDB. On demand, the collector queries any configured threat-intelligence and reputation sources (VirusTotal, AbuseIPDB, GreyNoise, AlienVault OTX, ThreatFox, IPQualityScore, Pulsedive, IPinfo, ip-api, RIPEstat) concurrently. Missing credentials, provider errors, rate limits, and disabled enrichment can produce partial or absent results. Returned evidence is cached (in memory and in MongoDB) and throttled by a shared outbound rate limiter; a background worker attempts to refresh recent hosts through available keyless sources. Every API key stays server-side and never reaches the browser. Contributing sources and the last-enrichment time are recorded so the UI can label reputation and CVE data as third-party associations rather than verified facts.

Build & packaging

The product ships as a small container built with a multi-stage Dockerfile: stage one uses node:22-alpine to build the browser bundle, SSR bundle, and route-specific HTML; stage two uses golang:1.26-alpine to embed that build and compile the vibescan andmigrate binaries (CGO_ENABLED=0 -trimpath); the final stage is a minimal alpine image (~60 MB) that runs as a non-root user. The scanner agent has its own image. Building the UI and Go binary in one image guarantees the embedded assets always match the code.

Hosting & infrastructure (AWS)

• Web host: an AWS EC2 t3.micro that pulls the image from Amazon ECR — the image is built elsewhere because the tiny host can't compile it.
• Reverse proxy: Caddy fronts the Go app, terminating TLS with automatic Let's Encrypt certificates and applying the security headers (HSTS, a Content-Security-Policy, frame-ancestor and MIME-sniffing protection, a strict referrer policy, and a restrictive permissions policy).
• Database: MongoDB Atlas M0 in the same region to keep latency low.
• Screenshots: a private S3 bucket served publicly through CloudFront (or Cloudflare R2), so image bytes never transit the app.
• Scanner: one low-rate host on an ordinary connection; migration to a dedicated VPS is planned.

CI/CD & operations

Continuous integration (GitHub Actions) gates every change on go vet, race-enabled go test, govulncheck, UI lint/tests/build, CodeQL, and a pinned Trivy dependency/secret scan. Deployment is deliberately locked down:

• Build the image, push it to ECR, then roll the EC2 host via AWS SSM Run Command. Automated deployments do not require SSH; separately allowlisted administrator SSH may be used for initial setup or maintenance.
• GitHub Actions uses OIDC to assume a scoped IAM role, so no long-lived AWS deployment key is stored in GitHub. The collector still uses narrowly scoped object-storage credentials from its protected server environment.
• A fast test job gates the deploy; a failed database migration fails the deploy; and a commit-identity or prerendered-page mismatch automatically rolls back to the previous image.

Two background workers keep the data fresh: an enrichment worker that keeps recent hosts cross-referenced, and a daily rollup worker that snapshots aggregate sample statistics once a day so exposure can be charted over time. Full details are in the methodology; the code is publicly viewable, with its current rights status documented in the repository.

Why it's built this way

The recurring theme is doing the boring, resilient thing so a small system stays trustworthy:

• One binary, same origin — fewer moving parts to deploy, secure, and reason about; production does not depend on cross-origin browser access.
• Fail soft — disk buffering, idempotent writes, bounded concurrency, and rate limits mean transient failures degrade gracefully instead of losing data or falling over.
• Make isolation deployable — the scanner ships independently from the collector so browser rendering and scan traffic can move to a dedicated VPS without changing the public service.
• Keep secrets server-side — every third-party key stays in the collector; the browser receives provider results and derived summaries, never credentials.
• Reduce deployment exposure — automated rollouts use SSM and short-lived OIDC credentials, are test-gated, and roll back unless the running commit and embedded UI both match the release.

Stack at a glance