How to Find All URLs on a Website: 6 Free Methods

Finding every URL on a website sounds simple until two tools return two different lists. A sitemap may contain pages that are no longer linked. A crawler may miss orphan pages. Google may index URLs that neither list makes obvious. The right method depends on which version of “all URLs” you actually need.

If you want a fast inventory of pages reachable through a site's internal links, start with the free Website URL Extractor. It follows internal links, records HTTP status codes and crawl depth, and lets you export the results. The methods below explain what that crawl can find, what it can miss, and how to cross-check it.

Quick answer: the best way to find all website URLs

For most audits, use three sources together:

1. Crawl the website to find pages users and search engines can reach through links.
2. Extract the XML sitemap to find URLs the site owner has declared.
3. Check Google Search Console to see which URLs Google has discovered or indexed.

No single source is guaranteed to contain every possible URL. Comparing these lists is more useful than trusting one list in isolation.

Method	Best for	Main limitation
Website crawl	Reachable pages, status codes and click depth	Can miss orphaned, blocked or JavaScript-only URLs
XML sitemap	Owner-declared canonical pages	May be incomplete or outdated
Google site: search	A quick sample of indexed pages	Not a complete or exportable index report
Google Search Console	Your own site's Google discovery and indexing data	Requires verified access
CMS or database export	Authoritative URLs stored by your own platform	May include drafts, duplicates or non-public records
Command-line crawl	Repeatable and customizable technical work	Requires setup and careful crawl rules

Method 1: Crawl the website

A website crawler begins with one URL, downloads its HTML and follows eligible internal links. It repeats that process until it reaches a page limit, crawl-depth limit or another scope rule.

This is the best method when you want to understand the site visitors and search crawlers can actually navigate. A crawl can reveal:

• Live internal page URLs
• Redirects and broken pages
• The page where each URL was discovered
• How many clicks a page is from the starting URL
• Images, stylesheets and scripts referenced across the site

To run a crawl without installing desktop software, open the Website URL Extractor, enter the homepage, choose a page limit and depth, and start the scan. Keep “Respect robots.txt” and the polite delay enabled unless you own the target and have a specific reason to change the crawl behavior.

Why a crawl may not find every URL

A crawler only knows about URLs it is given or can discover. It may miss:

• Orphan pages with no internal links
• Pages beyond the selected depth or page limit
• URLs blocked by robots.txt
• Pages available only after signing in
• Links created after client-side JavaScript runs
• URLs generated only after a form submission or user action

That is why a crawl should be compared with a sitemap and first-party data when completeness matters.

Method 2: Extract URLs from the XML sitemap

Most sites publish an XML sitemap at a location such as:

https://example.com/sitemap.xml
https://example.com/sitemap_index.xml

The site's robots.txt file may also declare the sitemap location. A sitemap is usually the fastest source for the canonical pages the owner wants search engines to discover. Google's sitemap guidance recommends listing the canonical URLs you want shown in search results.

Use the Sitemap URL Extractor to read a sitemap or nested sitemap index and export its URLs. This works well for content inventories, migration planning and checking whether newly published pages were added to the sitemap.

A sitemap is not proof that every listed page is healthy or internally linked. It can contain redirects, errors, non-canonical URLs or pages that have become orphaned. Compare the sitemap list with crawl results to find those differences.

Method 3: Use a Google site: search

For a quick check, search Google for:

site:example.com

You can narrow the query to a section or keyword:

site:example.com/blog/
site:example.com "pricing"

This is useful for spot checks: it can show whether a section appears in Google and may uncover unexpected indexed URLs. It is not a complete list of everything Google knows about the domain, and the displayed result count should not be treated as an exact audit total.

Use this method for quick research, not as the only source for a migration or deletion project.

Method 4: Check Google Search Console

If you own or manage the website, Google Search Console provides better first-party information than a public site: query.

Useful reports include:

• Page indexing: URLs Google has indexed, excluded or encountered with problems
• Search performance: Pages receiving impressions and clicks
• Sitemaps: URLs submitted through your sitemap files
• URL Inspection: Google's current information about one specific URL

Export the relevant reports and compare them with the crawler and sitemap inventories. For example, an indexed URL absent from the current crawl may be orphaned, linked externally or left over from an older site structure.

Search Console still does not represent every URL stored by your application. Drafts, private pages and URLs Google has never discovered belong in a CMS or database export instead.

Method 5: Export URLs from the CMS or database

When auditing your own website, the content management system can provide the closest thing to an authoritative content inventory.

Depending on the platform, export:

• Published posts and pages
• Products and category pages
• Media attachment pages
• Landing pages created by plugins or page builders
• Custom post types or database records with public routes

Clean the export before using it. CMS data can contain drafts, revisions, deleted records, duplicate URL variants and content that is not publicly accessible. Validate exported URLs against live HTTP responses and canonical tags.

This method is especially useful before a redesign or domain migration because it can expose stored pages that are no longer reachable through navigation.

Method 6: Use command-line tools

Developers may prefer a repeatable command-line crawl. Tools such as wget can recursively follow links, while custom scripts can fetch pages and parse anchors with precise rules.

A basic crawl might look like this:

wget --spider --recursive --no-parent https://example.com/

Be conservative with concurrency and delays. Respect the target's crawl rules, avoid authenticated or destructive URLs, and do not run an aggressive crawler against a site you do not control.

Command-line output also needs normalization. Fragments, tracking parameters, trailing slashes, redirects and canonical URLs can make the same page appear under multiple addresses.

How to combine the lists

After collecting URLs, normalize them before comparing sources:

1. Resolve relative paths into absolute URLs.
2. Remove URL fragments such as #features.
3. Decide how to handle tracking parameters and query strings.
4. Follow redirects and record the final destination.
5. Compare canonical URLs consistently.
6. Remove exact duplicates without losing the original source of each URL.

Then look for meaningful differences:

• In sitemap, not crawl: possibly orphaned, blocked or beyond the crawl limit
• In crawl, not sitemap: possibly a valid page missing from the sitemap, or a URL that should not be indexable
• In Search Console, not current site: possibly an old URL that needs a redirect or removal decision
• In CMS, not live: possibly a draft, deleted record or routing problem

The goal is not always to produce one enormous list. It is to understand why each source contains the URLs it does.

Which method should you choose?

Use a website crawl when you need reachable URLs, status codes and site depth. Use an XML sitemap when you need the owner's declared indexable URLs. Use Search Console for Google's view of a site you control, and use the CMS or database when you need the underlying content inventory.

For one webpage rather than an entire site, use the Webpage Link Extractor. It returns internal links, external links and resource URLs from that single page without recursively crawling the domain.

For a whole-site inventory, run the Website URL Extractor, export the results, and compare them with the site's sitemap. That combination catches far more useful discrepancies than any one method alone.

Frequently asked questions

Can I find every URL on a website?

You can build a strong inventory by combining a crawl, sitemap, Search Console and CMS data, but no public crawler can guarantee knowledge of every URL a server could generate. Private, orphaned, parameterized and dynamically generated URLs may remain undiscovered.

What is the fastest way to get a website URL list?

Check the XML sitemap when you want the owner's declared URL list. Run a website crawl when you want pages that are actually reachable through internal links. Both can be exported for comparison.

Is a sitemap a complete list of website pages?

Not necessarily. A sitemap contains URLs selected by the site owner or CMS. It may omit pages, include outdated URLs or list pages that are no longer internally linked.

Can Google site: search show all indexed pages?

No. It is useful for discovery and spot checks, but it should not be treated as a complete export of Google's index. Website owners should use Search Console for more dependable indexing information.

What is the difference between a link extractor and a website crawler?

A link extractor reads one page and lists the links present in that page's HTML. A website crawler starts with one page and continues following eligible internal links across the site.

Can I use a crawled URL list to create a sitemap?

Yes, but review the list first. A production sitemap should contain canonical, indexable URLs you want search engines to discover—not every URL the crawler happened to find.