What Open-Source SEO Crawlers Check

When teams ask “what should we crawl before we rewrite content?”, open-source SEO spiders are a useful reference—not because any tool invents ranking rules, but because they make a long checklist concrete. This note records design facts observed from the public repository puneetindersingh/open-seo-crawler (MIT). It is Mika’s rewrite of those observations for audit practice. We are not republishing that project’s documentation.

Why study a crawler’s checklist

A crawler’s value for auditors is the shape of its inventory. Open SEO Crawler presents itself as a self-hosted spider: concurrent fetches, optional JavaScript rendering, CMS-aware presets, and bulk exports. The interesting part for Mika is not the installer—it is which failure modes the tool chooses to name, group, and export.

Those named checks become a translation layer between engineering language (“status codes, canonicals, soft 404s”) and buyer-facing risk (“this page may never be discovered cleanly”). An audit worksheet that mirrors those categories is easier to defend than a vague “SEO health” score.

Per-page signals the spider surfaces

From public feature descriptions, each crawled HTML page is assessed along familiar on-page axes:

• Titles and meta descriptions — missing, too short, too long, or duplicated across URLs.
• H1 structure — missing H1, multiple H1s, or H1 identical to the title.
• Canonical tags — missing, self-referencing, or pointing elsewhere (“canonicalised”).
• Indexability — noindex via meta robots or X-Robots-Tag.
• Hreflang — extraction plus validation, including missing return tags.
• Structured data presence — JSON-LD and microdata types detected (presence, not a ranking promise).
• Open Graph / Twitter cards — common social metadata gaps.
• Thin or near-duplicate bodies — short word counts and shingle-based similarity pairs.
• Redirects and chains — multi-hop paths with hop lists; normalisations separated from “real” redirects.
• HTTP errors — 4xx / 5xx with retry context.
• Images missing alt, mixed content, security headers, viewport presence, and deep click-distance pages.

None of these items “guarantee” citations or rankings. They do tell an auditor whether the site’s HTML is internally consistent enough to be crawled and understood without guessing.

Sitewide reports: sitemap, duplicates, broken links

Beyond single-page rows, the project emphasises cross-URL reports. Publicly described bulk views include all titles / metas / H1s / canonicals, duplicate groupings, redirect chains, response-code mixes, deep pages, images missing alt, and hreflang consistency. Sitemap analysis compares the crawl set against sitemap.xml: URLs missing from the sitemap, sitemap-only URLs, non-200 entries, and redirects inside the sitemap.

Broken-link and malformed-href handling is treated as first-class hygiene. Scheme-less or plain-text href values (for example, an address pasted into a link) are reported as malformed instead of exploding into phantom 404s under every page that shares a footer. That design choice matters for B2B sites with shared chrome: one bad footer link should not look like hundreds of unique failures.

Crawl budget and soft-404 traps

Another observable theme is protecting the crawl from infinite or low-value URL spaces. The public materials describe skipping non-page paths, repeating path segments, common page-builder pagination parameters, and post-crawl probes that test whether the server returns 200 for URLs that cannot exist. A separate crawl-budget view buckets query parameters and can suggest robots Disallow lines for faceted or tracking traps.

For auditors, the lesson is procedural: inventory phantom URL classes before you invent new content. Expanding topic coverage on a site that soft-404s arbitrary paths will waste crawl attention and muddy diagnostics.

What this suggests for website audits

Mika’s takeaway is to treat open crawler checklists as a coverage map, not a scoreboard:

1. Separate discovery health (status codes, robots, sitemap diffs, soft 404s) from page clarity (titles, H1, thin copy).
2. Export duplicate and hreflang reports before rewriting multilingual templates.
3. Fix shared-template defects (footer links, missing alts on repeated assets) once, then re-crawl.
4. Use JS-vs-no-JS comparison—when available—to see what non-executing crawlers actually receive.

That sequence keeps humans in control: the tool lists evidence; people decide priority against business pages that matter.

What this note is not

This article does not install, fork, or mirror Open SEO Crawler. It does not claim Mika copied their docs. MIT here means we may study and attribute facts; our prose remains Mika’s. We also do not promise that clearing every crawler warning will produce AI citations or organic growth. Technical cleanliness is necessary hygiene—not a substitute for source-backed answers buyers need.