RizCraft Logo RizCraft

Website URL Extractor

Scan a website by following its internal links. Find reachable page URLs, HTTP status codes, crawl depth and source pages, then export the results.

Need links from one webpage?

Crawl settings

Free limit: 250 pages per crawl.

0 is the start page. Depth 2 follows links up to two clicks away.

Options
Advanced: include / exclude paths
Pages that build their links with JavaScript can't be crawled yet.

Similar SEO Tools

See All
XML Sitemap Generator
Generate a valid sitemap.xml file from a list of URLs instantly
Robots.txt Generator
Build a valid robots.txt file with crawler rules, AI bot blocking, and sitemap links
Meta Tags Checker
Check, edit, and preview website meta tags for social media

Related Guides

How to Find All URLs on a Website: 6 Free Methods

Find every useful URL on a website by combining a live crawl, XML sitemap, Google Search, Search Console, CMS exports and command-line tools.

Extract All Reachable URLs From a Website

The Website URL Extractor scans a public site and builds a structured inventory of URLs reachable from the starting page. It follows eligible internal links like a lightweight website crawler, records where each URL was found, and reports the status and click depth of pages it visits. Use it when a single-page link list is not enough and you need to discover content across a domain.

Set a crawl depth, scan up to 250 pages, keep the crawl on one domain or include subdomains, and optionally seed discovery from sitemap.xml. Include and exclude path rules help focus a scan on sections such as /blog/ or avoid areas such as carts, account pages and faceted URLs.

What the Website Scan Returns

Each crawled page includes its final URL, HTTP status, crawl depth and the page where it was discovered. The results also separate page links, images, CSS and JavaScript resources so you can inspect the site without mixing every resource into one list.

  • Reachable page URLs: Build an inventory for migrations, content audits, information-architecture reviews and sitemap planning.
  • HTTP status results: Separate successful pages, redirects and errors instead of treating every discovered URL as healthy.
  • Depth and discovery source: See how many clicks a page is from the start URL and which page linked to it.
  • Resource groups: Review discovered links, images, stylesheets and scripts in separate result tabs.
  • Reusable exports: Download filtered data as CSV, TXT or JSON, or create an XML sitemap draft from successful pages.

How to Get a List of URLs From a Website

Not sure whether you should crawl the site, read its sitemap or check Google's index? See our complete guide to finding all URLs on a website for six methods and guidance on when to use each one.

  1. Enter the homepage or section URL where discovery should begin.
  2. Choose the maximum number of pages, crawl depth and domain scope.
  3. Keep robots.txt and the polite delay enabled for a respectful scan, then add any path filters you need.
  4. Start the crawl and monitor discovered, scanned and error counts as it runs.
  5. Filter the results by URL, status or depth, then export a list or sitemap draft.

Website URL Extractor vs. Link Extractor

The Webpage Link Extractor reads one URL and lists every link or selected resource found in that page's HTML. The Website URL Extractor starts with one URL but continues to visit eligible internal pages. Choose the first tool for a fast one-page inventory and this tool for site discovery, crawl depth, status reporting and sitemap preparation.

Website Crawl vs. XML Sitemap

A crawl shows what can be reached by following links under the selected rules. An XML sitemap shows URLs the site owner has declared, including pages that may not be linked internally. Comparing the two can reveal sitemap URLs that are hard to reach and linked pages that are missing from the sitemap. Enable sitemap seeding to combine both discovery sources, or use the Sitemap URL Extractor when you only need to read an existing sitemap file.

What the Scan Cannot Guarantee

No crawler can infer every URL that exists on a server. Orphan pages, private or authenticated content, blocked paths, form-generated URLs, JavaScript-only navigation and pages beyond the chosen limit may not be discovered. A completed scan therefore represents URLs reachable from the selected starting point under the selected settings—not a guarantee of every possible URL on the domain.

Frequently Asked Questions

How does the Website URL Extractor find pages?
It starts with the URL you enter and follows eligible internal links up to the selected depth and page limit. Optional sitemap seeding can add URLs declared in sitemap.xml.
Will it find every URL on a website?
It finds URLs reachable within your chosen crawl settings. Orphan pages, blocked paths, JavaScript-only links and pages beyond the page or depth limit may not appear. The results clearly identify when a scan stops at a configured limit.
Can I use the results to build a sitemap?
Yes, you can download a sitemap draft containing successful page URLs. Review canonical URLs, noindex directives and other indexing rules before publishing it as your production sitemap.
Does the scan respect robots.txt?
Yes, when the “Respect robots.txt” option is enabled. The scanner also observes supported crawl-delay rules and uses a polite request delay by default.
What is crawl depth?
Depth measures link distance from the starting URL. The start page is depth 0, its direct internal links are depth 1, and pages linked from those pages are depth 2. A higher depth reaches farther but can take longer and consume the page limit sooner.
Can the scan include subdomains?
Yes. Choose “Include subdomains” when URLs such as docs.example.com or shop.example.com belong in the same inventory. Keep “This domain only” selected when you want the crawl restricted to the starting host.
Why are some pages listed as discovered but not scanned?
A URL can be discovered near the end of a crawl after the maximum page count or depth has already been reached. Scope rules, include or exclude paths, robots.txt and asset skipping can also prevent a discovered URL from being fetched.
Can it crawl JavaScript-rendered or private pages?
The scanner reads the HTML returned for public URLs and does not sign in or operate an interactive browser. Links created only after JavaScript runs and pages that require authentication may not be found.
What can I export after a website scan?
Download crawl results as CSV, TXT or JSON. You can also create an XML sitemap draft from successful page URLs; review canonical, noindex and content-quality decisions before submitting that file to search engines.