How to Pull Every Link Off a Webpage (Not Just the Sitemap)
2026-09-11
A sitemap tells you which pages a site wants indexed. It doesn't tell you what those pages actually link to — footer links, in-content references, an old partnership page nobody remembers linking, a broken outbound link from three years ago. For that, you need to read the page itself, not the XML file describing it.
Why this is a different job than the sitemap extractor
(We covered pulling URLs from a sitemap.xml in an earlier post — link it here once it's live.) That tool reads a site's own declared list of pages. This one does the opposite: point it at a single live URL, and it fetches that page and lists every link actually present in its HTML, internal and external.
The two catch different things. A sitemap can list a page that no longer links to anything relevant — orphaned in practice, indexed in theory. A live page can link to something the sitemap never mentions at all. If you're doing a real content audit, you usually want both, not just one.
What people actually use this for
Broken link audits. Pull every link off a page, then run the list through a link checker to find 404s. Doing this manually by clicking every link on a long page eats an afternoon for no reason.
Competitor research. A competitor's linking pattern tells you things their content doesn't say directly — who they partner with, what tools their own site runs on (footer credits, embedded widgets), which of their own pages they consider important enough to link internally.
Migration and redesign audits. Before a redesign, extract the link inventory of key pages to make sure nothing important gets silently dropped when the new template goes live. It's easy to lose a legacy link during a rebuild without anyone noticing until a partner emails asking why their listing disappeared.
Backlink verification. If someone tells you they've linked to your site, paste their page in and check — faster than scanning a link-heavy page by eye.
One honest limitation
The Website URL Extractor reads the HTML the server sends back. If a page builds its navigation or links entirely in JavaScript after the initial load — some single-page apps do this — a link that only appears after client-side rendering may not show up in the extracted list. Most content sites, blogs, and marketing pages render their links directly in the HTML and won't have this problem; heavily JS-driven web apps sometimes will. If a result looks shorter than you'd expect for a link-heavy page, that's the first thing to check.
Run both together
Extract from the site's sitemap.xml, then run this against a handful of its most important pages directly. Where the two lists diverge is often where the interesting findings are — an important page missing from the sitemap, or a page linking somewhere the sitemap gives no indication of.